[alert-handler] auto-fix Nvidia GPU low performance issue

[suiguoxin]

No description provided.

[coveralls]

Coverage remained the same at 34.02% when pulling 9f24d6e on suiguoxin:gpu-perf into fe18fd9 on microsoft:master.

[Binyang2014]

Please do some tests in INT bed. Seems the script doesn't work

[suiguoxin]

Test cases:

• enable this feature by adding the following customized route & receiver in services-configuration.yaml:

customized-routes:
  routes:
  - receiver: pai-email-admin-and-fix-nvidia-gpu-low-perf
    match:
      alertname: NodeGpuLowPerfState
customized-receivers: # receivers are combination of several actions
- name: "pai-email-admin-and-fix-nvidia-gpu-low-perf"
  actions:
    email-admin:
    fix-nvidia-gpu-low-perf:

• manully send an alert by POST to https://xxx.openpai.org/alert-manager/api/v1/alerts with the following body:

[
        {
            "labels": {
                "alertname": "NodeGpuLowPerfState",
                "minor_number": "0",
                "node_name": "node6",
                "severity": "warn"
            },
            "generatorURL": "alert/script",
            "fingerprint": "6b8102e96c9e6b2a"
        }
]

• check k8s job named nvidia-gpu-low-perf-fixer-xxx and related pods created, check logs, the GPU state & clocks should been changed