Add / Remove Nodes with `layout.yaml`

[suiguoxin]

Target

allow users to add / remove nodes with layout.yaml instead of hosts.yml.

Design

add nodes :

Step 1: add nodes info to layout.yaml

pull layout to local

./paictl.py config pull -o /cluster-configuration

add the corresponding nodes in to the machine-list field in layout.yaml

machine-sku:
  xxx-sku: 
    mem: ...
    cpu: ...

machine-list:
  - hostname: a
    hostip: x.x.x.x
    machine-type: xxx-sku
    pai-worker: "true"
  - hostname: b
    hostip: x.x.x.x
    machine-type: xxx-sku
    pai-worker: "true"

push layout back to k8s

./paictl.py config push -p /cluster-configuration/ -m service

Step 2: add nodes to k8s

./paictl.py node add -n a b

in this step, we will:

• generate hosts.yaml, openpai.yml with layout.yaml, config.yaml
• call ansible-playbook to add the nodes to k8s

Step 3: restart related services

Modify HiveD config in services-configuration.yaml manually, restart services: cluster-configuration hivedscheduler rest-server

remove nodes:

Step 1: remove nodes from k8s

./paictl.py node remove --nodes a b

in this step, we will:

• generate hosts.yaml, openpai.yml with layout.yaml, config.yaml
• call ansible-playbook to remove the nodes from k8s
• modify layout.yaml in the k8s

Step 2: restart related services

Modify HiveD config in services-configuration.yaml, restart services: cluster-configuration hivedscheduler rest-server

Problems:

• config.yaml is local but layout.yaml is saved in k8s

[hzy46]

scale to add-nodes ? Also remove-nodes is better in my view.

[suiguoxin]

scale to add-nodes ? Also remove-nodes is better in my view.

After discussion, we'll use ./paictl.py node add -n a b & ./paictl.py node remove -n a b

[suiguoxin]

Implemented in #5400