Cordon node

[suiguoxin]

Cordon node when NvidiaSmiEccError:

• collect node_name in job_exporter
• set alert in Prometheus
• add cordon-nodes action in alert-manager, cordon nodes through k8s api patchNode

Refer to Issue #4789

[coveralls]

Coverage decreased (-0.1%) to 34.276% when pulling de9c433 on suiguoxin:cordon-node into cf4e6a8 on microsoft:master.

[fanyangCS]

Maybe you could also implement uncordon as well?

[Binyang2014]

Maybe you could also implement uncordon as well?

I suggest uncordon node manually. I think alert resolve message may be false positive. Such as job-exporter trigger the alert, then the job exporter is crash. Since there is no metric anymore, the alert rules is not satisfied and the alert maybe resolved by Prometheus. But error still here.

So I suggest admin uncordon the node manually when he make sure everything is OK

[suiguoxin]

Maybe you could also implement uncordon as well?

I suggest uncordon node manually. I think alert resolve message may be false positive. Such as job-exporter trigger the alert, then the job exporter is crash. Since there is no metric anymore, the alert rules is not satisfied and the alert maybe resolved by Prometheus. But error still here.

So I suggest admin uncordon the node manually when he make sure everything is OK

@fanyangCS @Binyang2014 Added a brief instruction for admin about the alert and how to uncordon nodes here.
https://github.com/suiguoxin/pai/blob/cordon-node/docs/manual/cluster-admin/troubleshooting.md#nvidiasmieccerror