[display event] add event watcher in database controller

[hzy46]

database size control strategy:

• In event watcher, check the disk usage in the beginning. If the disk usage > 80%, stop the event watcher and exit with a non-zero code.
• The disk check also happens every 60s. If the disk usage > 80%, stop the event watcher and exit with a non-zero code.
• 60s and 80% are configurable.

Problem found:

• If a job generates too many events, it will affect all other jobs.

[coveralls]

Coverage remained the same at 34.383% when pulling 87d4225 on zhiyuhe/add_event_watcher into 9755553 on master.

[yqwang-ms]

Better to limit the quota per job and global, and does not impact the critical path.

[hzy46]

Recorded in #4953 . We can solve this problem in the future.

[yqwang-ms]

Also limit for history ? Why not move non-critical things to another DB server?

[hzy46]

Recorded in #4954 . We can solve this problem in the future.

[hzy46]

Event Watcher Test Cases:

1. Test: the event watcher works properly

Submit a job that will be always in waiting status (e.g. use a lot of resource). Then check if there is any event about "failed scheduling" on the job event page after a few minutes.

2. Test: the event watcher can handle a large number of events.

Submit a job with 2000+ tasks. After a few minutes check the event page can work properly.

3. Test: the event watcher will exit if too much disk size is used.

Go to internal storage to see the existing usage:

kubectl exec -it `kubectl get po  | grep internal-storage | awk '{print $1}' ` bash
df -h

Please notice the usage of loop device /paiInternal/storage.

Create a big file under /paiInternal/storage and make its usage larger than 80%.

After a few minutes, confirm that 1. there is a NodeFilesystemUsage alert shown on webportal 2. the event watcher should exit automatically.

Remove the big file. After a few minutes, confirm that: 1. there is no more NodeFilesystemUsage alert 2. the event watcher should work properly, and we can see events of new jobs on webportal.