Enhance/cache pd status by 9547 · Pull Request #986 · pingcap/tiup

9547 · 2020-12-12T16:19:49Z

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

The old one need ~2minutes

root@control:/tiup-cluster# time bin/tiup-cluster display c1
Cluster type:    tidb
Cluster name:    c1
Cluster version: v4.0.4
SSH type:        builtin

ID                  Role            Host          Ports                            OS/Arch       Status  Data Dir                                                 Deploy Dir
--                  ----            ----          -----                            -------       ------  --------                                                 ----------
172.19.0.101:9093   alertmanager    172.19.0.101  9093/9094                        linux/x86_64  Up      /home/tidb/deploy/alertmanager-9093/data                 /home/tidb/deploy/alertmanager-9093
172.19.0.103:8300   cdc             172.19.0.103  8300                             linux/x86_64  Down    -                                                        /home/tidb/deploy/cdc-8300
172.19.0.104:8300   cdc             172.19.0.104  8300                             linux/x86_64  Down    -                                                        /home/tidb/deploy/cdc-8300
172.19.0.105:8300   cdc             172.19.0.105  8300                             linux/x86_64  Down    -                                                        /home/tidb/deploy/cdc-8300
172.19.0.101:8249   drainer         172.19.0.101  8249                             linux/x86_64  Up      /home/tidb/data/drainer-8249/data                        /home/tidb/deploy/drainer-8249
172.19.0.101:3000   grafana         172.19.0.101  3000                             linux/x86_64  Up      -                                                        /home/tidb/deploy/grafana-3000
172.19.0.103:2379   pd              172.19.0.103  2379/2380                        linux/x86_64  Down    /home/tidb/deploy/pd-2379/data                           /home/tidb/deploy/pd-2379
172.19.0.104:2379   pd              172.19.0.104  2379/2380                        linux/x86_64  Down    /home/tidb/deploy/pd-2379/data                           /home/tidb/deploy/pd-2379
172.19.0.105:2379   pd              172.19.0.105  2379/2380                        linux/x86_64  Down    /home/tidb/deploy/pd-2379/data                           /home/tidb/deploy/pd-2379
172.19.0.101:9090   prometheus      172.19.0.101  9090                             linux/x86_64  Up      /home/tidb/deploy/prometheus-9090/data                   /home/tidb/deploy/prometheus-9090
172.19.0.102:9090   prometheus      172.19.0.102  9090                             linux/x86_64  Up      /home/tidb/deploy/prometheus-9090/data                   /home/tidb/deploy/prometheus-9090
172.19.0.103:8250   pump            172.19.0.103  8250                             linux/x86_64  Up      /home/tidb/deploy/pump-8250/data                         /home/tidb/deploy/pump-8250
172.19.0.104:8250   pump            172.19.0.104  8250                             linux/x86_64  Up      /home/tidb/deploy/pump-8250/data                         /home/tidb/deploy/pump-8250
172.19.0.105:8250   pump            172.19.0.105  8250                             linux/x86_64  Up      /home/tidb/deploy/pump-8250/data                         /home/tidb/deploy/pump-8250
172.19.0.101:4000   tidb            172.19.0.101  4000/10080                       linux/x86_64  Up      -                                                        /home/tidb/deploy/tidb-4000
172.19.0.102:4000   tidb            172.19.0.102  4000/10080                       linux/x86_64  Up      -                                                        /home/tidb/deploy/tidb-4000
172.19.0.103:9000   tiflash         172.19.0.103  9000/8123/3930/20170/20292/8234  linux/x86_64  Down    /home/tidb/deploy/tiflash-9000/data1,/data/tiflash-data  /home/tidb/deploy/tiflash-9000
172.19.0.101:20160  tikv            172.19.0.101  20160/20180                      linux/x86_64  Down    /home/tidb/deploy/tikv-20160/data                        /home/tidb/deploy/tikv-20160
172.19.0.103:20160  tikv            172.19.0.103  20160/20180                      linux/x86_64  Down    /home/tidb/my_kv_data                                    /home/tidb/deploy/tikv-20160
172.19.0.104:20160  tikv            172.19.0.104  20160/20180                      linux/x86_64  Down    /home/tidb/deploy/tikv-20160/data                        /home/tidb/deploy/tikv-20160
172.19.0.105:20160  tikv            172.19.0.105  20160/20180                      linux/x86_64  Down    /home/tidb/deploy/tikv-20160/data                        /home/tidb/deploy/tikv-20160
172.19.0.103:7077   tispark-master  172.19.0.103  7077/8080                        linux/x86_64  Up      -                                                        /home/tidb/deploy/tispark-master-7077
172.19.0.104:7078   tispark-worker  172.19.0.104  7078/8081                        linux/x86_64  Up      -                                                        /home/tidb/deploy/tispark-worker-7078
Total nodes: 23

real    1m56.029s
user    0m0.287s
sys     0m0.133s

After cached, the used time is limited in 3*10s

root@control:/tiup-cluster# time bin/tiup-cluster display c1
Cluster type:       tidb
Cluster name:       c1
Cluster version:    v4.0.4
SSH type:           builtin
ID                  Role            Host          Ports                            OS/Arch       Status  Data Dir                                                 Deploy Dir
--                  ----            ----          -----                            -------       ------  --------                                                 ----------
172.19.0.101:9093   alertmanager    172.19.0.101  9093/9094                        linux/x86_64  Up      /home/tidb/deploy/alertmanager-9093/data                 /home/tidb/deploy/alertmanager-9093
172.19.0.103:8300   cdc             172.19.0.103  8300                             linux/x86_64  Down    -                                                        /home/tidb/deploy/cdc-8300
172.19.0.104:8300   cdc             172.19.0.104  8300                             linux/x86_64  Down    -                                                        /home/tidb/deploy/cdc-8300
172.19.0.105:8300   cdc             172.19.0.105  8300                             linux/x86_64  Down    -                                                        /home/tidb/deploy/cdc-8300
172.19.0.101:8249   drainer         172.19.0.101  8249                             linux/x86_64  Up      /home/tidb/data/drainer-8249/data                        /home/tidb/deploy/drainer-8249
172.19.0.101:3000   grafana         172.19.0.101  3000                             linux/x86_64  Up      -                                                        /home/tidb/deploy/grafana-3000
172.19.0.103:2379   pd              172.19.0.103  2379/2380                        linux/x86_64  Down    /home/tidb/deploy/pd-2379/data                           /home/tidb/deploy/pd-2379
172.19.0.104:2379   pd              172.19.0.104  2379/2380                        linux/x86_64  Down    /home/tidb/deploy/pd-2379/data                           /home/tidb/deploy/pd-2379
172.19.0.105:2379   pd              172.19.0.105  2379/2380                        linux/x86_64  Down    /home/tidb/deploy/pd-2379/data                           /home/tidb/deploy/pd-2379
172.19.0.101:9090   prometheus      172.19.0.101  9090                             linux/x86_64  Up      /home/tidb/deploy/prometheus-9090/data                   /home/tidb/deploy/prometheus-9090
172.19.0.102:9090   prometheus      172.19.0.102  9090                             linux/x86_64  Up      /home/tidb/deploy/prometheus-9090/data                   /home/tidb/deploy/prometheus-9090
172.19.0.103:8250   pump            172.19.0.103  8250                             linux/x86_64  Up      /home/tidb/deploy/pump-8250/data                         /home/tidb/deploy/pump-8250
172.19.0.104:8250   pump            172.19.0.104  8250                             linux/x86_64  Up      /home/tidb/deploy/pump-8250/data                         /home/tidb/deploy/pump-8250
172.19.0.105:8250   pump            172.19.0.105  8250                             linux/x86_64  Up      /home/tidb/deploy/pump-8250/data                         /home/tidb/deploy/pump-8250
172.19.0.101:4000   tidb            172.19.0.101  4000/10080                       linux/x86_64  Up      -                                                        /home/tidb/deploy/tidb-4000
172.19.0.102:4000   tidb            172.19.0.102  4000/10080                       linux/x86_64  Up      -                                                        /home/tidb/deploy/tidb-4000
172.19.0.103:9000   tiflash         172.19.0.103  9000/8123/3930/20170/20292/8234  linux/x86_64  N/A     /home/tidb/deploy/tiflash-9000/data1,/data/tiflash-data  /home/tidb/deploy/tiflash-9000
172.19.0.101:20160  tikv            172.19.0.101  20160/20180                      linux/x86_64  N/A     /home/tidb/deploy/tikv-20160/data                        /home/tidb/deploy/tikv-20160
172.19.0.103:20160  tikv            172.19.0.103  20160/20180                      linux/x86_64  N/A     /home/tidb/my_kv_data                                    /home/tidb/deploy/tikv-20160
172.19.0.104:20160  tikv            172.19.0.104  20160/20180                      linux/x86_64  N/A     /home/tidb/deploy/tikv-20160/data                        /home/tidb/deploy/tikv-20160
172.19.0.105:20160  tikv            172.19.0.105  20160/20180                      linux/x86_64  N/A     /home/tidb/deploy/tikv-20160/data                        /home/tidb/deploy/tikv-20160
172.19.0.103:7077   tispark-master  172.19.0.103  7077/8080                        linux/x86_64  Up      -                                                        /home/tidb/deploy/tispark-master-7077
172.19.0.104:7078   tispark-worker  172.19.0.104  7078/8081                        linux/x86_64  Up      -                                                        /home/tidb/deploy/tispark-worker-7078
Total nodes: 23

real    0m26.010s
user    0m0.209s
sys     0m0.122s

One more thing

for the dm's cluster, I've highlighted the dm's status code, similar to tidb-cluster's, like below:

And if dm-masters were down, the dm-workers' status previous as below:

after this or, changed to: (Down -> N/A, means worker's status depends on master, but all masters were down)

Dashboard url

display with dashboard-url appended

root@control:/tiup-cluster# tiup-cluster display xxx
Cluster type:       tidb
Cluster name:       xxx
Cluster version:    v4.0.4
SSH type:           builtin
Dashboard URL:      http://n5:2379/dashboard
ID        Role            Host  Ports                            OS/Arch       Status  Data Dir                                  Deploy Dir
...

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Release notes:

* enhancement: speed up the `{tiup|dm} cluster display;
* feature: Add dashboard URL in `tiup cluster display`;
* feature: highlight dm's status in `dm cluster display`;

codecov-io · 2020-12-12T16:22:23Z

Codecov Report

Merging #986 (9590b25) into master (edb12b8) will increase coverage by 0.00%.
The diff coverage is 88.42%.

@@           Coverage Diff           @@
##           master     #986   +/-   ##
=======================================
  Coverage   55.64%   55.65%           
=======================================
  Files         279      279           
  Lines       19715    19743   +28     
=======================================
+ Hits        10970    10987   +17     
- Misses       7037     7044    +7     
- Partials     1708     1712    +4

Flag	Coverage Δ
cluster	`43.55% <85.55%> (-0.02%)`	⬇️
dm	`23.97% <55.78%> (+0.07%)`	⬆️
integrate	`49.92% <88.42%> (+0.01%)`	⬆️
playground	`20.32% <0.00%> (-0.01%)`	⬇️
tiup	`16.50% <0.00%> (+0.02%)`	⬆️
unittest	`23.01% <0.00%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/cluster/spec/util.go	`81.39% <0.00%> (-0.83%)`	⬇️
pkg/cluster/spec/spec.go	`88.57% <75.00%> (-0.40%)`	⬇️
pkg/cluster/spec/pd.go	`69.87% <77.77%> (-2.53%)`	⬇️
pkg/cluster/manager/display.go	`82.67% <91.30%> (+1.04%)`	⬆️
components/dm/spec/topology_dm.go	`82.40% <100.00%> (+0.56%)`	⬆️
pkg/cluster/api/pdapi.go	`60.06% <100.00%> (-1.86%)`	⬇️
pkg/utils/http_client.go	`67.56% <100.00%> (-4.66%)`	⬇️
pkg/repository/store/txn.go	`62.01% <0.00%> (+2.32%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update edb12b8...9590b25. Read the comment docs.

breezewish · 2020-12-14T04:53:07Z

Hi, could you update your PR description about the Dashboard URL change?

9547 · 2020-12-14T15:44:10Z

Hi, could you update your PR description about the Dashboard URL change?

Sorry, and I'll add it in the Release notes, it's anything else I'm missing?

breezewish · 2020-12-14T16:39:06Z

Hi, could you update your PR description about the Dashboard URL change?

Sorry, and I'll add it in the Release notes, it's anything else I'm missing?

Actually I only want to know what it looks like in the terminal output for this Dashbaord URL change :)

9547 · 2020-12-16T06:35:44Z

@breeswish PTAL

root@control:/tiup-cluster# tiup-cluster display xxx
Cluster type:       tidb
Cluster name:       xxx
Cluster version:    v4.0.4
SSH type:           builtin
Dashboard URL:      http://n5:2379/dashboard
ID        Role            Host  Ports                            OS/Arch       Status  Data Dir                                  Deploy Dir
...

lucklove · 2020-12-16T06:59:31Z

Maybe we can remove the --dashboard flag if we merged it into normal display? @breeswish

9547 · 2020-12-16T12:27:53Z

Maybe we can remove the --dashboard flag if we merged it into normal display? @breeswish

LGTM

lucklove · 2020-12-21T06:56:05Z

/lgtm

lucklove · 2020-12-21T06:56:16Z

/merge

ti-chi-bot · 2020-12-21T06:56:24Z

Can merge label has been added.

Details

Git tree hash: 9590b25

* typo(cluster/pdapi): misused arg name * enhance(cluster/pd): pd.Status check by itself * feat(cluster/spec): add GetDashardAddress * enhance(utils): set http clien't dial timeout to 5second * feat(cluster/manager): cache pd's status * feat(cluster/manager): display dashboard url * tests(cluster): add display result check * feat(dm): get master's status by local host * feat(cluster/manager): cache dm's master status * feat(cluster/manager): highlight display dm's status * feat(cluster/manager): highlight display dm's status * refact(cluster/pd): rm notused suffix * typo(spec/cluster): UP -> Up Co-authored-by: SIGSEGV <gnu.crazier@gmail.com>

ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 12, 2020

ti-chi-bot requested review from breezewish and lonng December 12, 2020 16:20

9547 added 7 commits December 13, 2020 08:00

typo(cluster/pdapi): misused arg name

4977354

enhance(cluster/pd): pd.Status check by itself

dcd2aa3

feat(cluster/spec): add GetDashardAddress

3c80976

enhance(utils): set http clien't dial timeout to 5second

644d28a

feat(cluster/manager): cache pd's status

8e86614

feat(cluster/manager): display dashboard url

fbea0c3

tests(cluster): add display result check

3c20b65

9547 force-pushed the feature/cache-pd-status branch from 895545e to 3c20b65 Compare December 13, 2020 00:01

9547 added 4 commits December 13, 2020 17:20

feat(dm): get master's status by local host

5e717d2

feat(cluster/manager): cache dm's master status

b05679e

feat(cluster/manager): highlight display dm's status

a4e841b

feat(cluster/manager): highlight display dm's status

9d72a8f

Merge branch 'master' into feature/cache-pd-status

d5edd29

Merge branch 'master' into feature/cache-pd-status

5594bcd

Merge branch 'master' into feature/cache-pd-status

6468fdd

AstroProfundis added the category/usability Categorizes issue or PR as a usability enhancement. label Dec 15, 2020

Merge branch 'master' into feature/cache-pd-status

79a3bee

Merge branch 'master' into feature/cache-pd-status

3168838

lucklove reviewed Dec 17, 2020

View reviewed changes

Comment thread pkg/cluster/spec/pd.go Outdated

refact(cluster/pd): rm notused suffix

5ce6a0c

9547 added 2 commits December 18, 2020 18:47

Merge branch 'master' into feature/cache-pd-status

b9a630f

typo(spec/cluster): UP -> Up

9590b25

9547 requested a review from lucklove December 19, 2020 03:38

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 21, 2020

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 21, 2020

ti-chi-bot merged commit aee2af0 into pingcap:master Dec 21, 2020

lucklove changed the title ~~Feature/cache pd status~~ Enhance/cache pd status Dec 31, 2020

lucklove added this to the v1.3.1 milestone Dec 31, 2020

9547 deleted the feature/cache-pd-status branch April 6, 2021 23:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance/cache pd status#986

Enhance/cache pd status#986
ti-chi-bot merged 19 commits intopingcap:masterfrom
jsvisa:feature/cache-pd-status

9547 commented Dec 12, 2020 •

edited

Loading

Uh oh!

codecov-io commented Dec 12, 2020 •

edited

Loading

Uh oh!

breezewish commented Dec 14, 2020

Uh oh!

9547 commented Dec 14, 2020

Uh oh!

breezewish commented Dec 14, 2020

Uh oh!

9547 commented Dec 16, 2020

Uh oh!

lucklove commented Dec 16, 2020

Uh oh!

9547 commented Dec 16, 2020

Uh oh!

Uh oh!

lucklove commented Dec 21, 2020

Uh oh!

lucklove commented Dec 21, 2020

Uh oh!

ti-chi-bot commented Dec 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

9547 commented Dec 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

One more thing

Dashboard url

Uh oh!

codecov-io commented Dec 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

breezewish commented Dec 14, 2020

Uh oh!

9547 commented Dec 14, 2020

Uh oh!

breezewish commented Dec 14, 2020

Uh oh!

9547 commented Dec 16, 2020

Uh oh!

lucklove commented Dec 16, 2020

Uh oh!

9547 commented Dec 16, 2020

Uh oh!

Uh oh!

lucklove commented Dec 21, 2020

Uh oh!

lucklove commented Dec 21, 2020

Uh oh!

ti-chi-bot commented Dec 21, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

9547 commented Dec 12, 2020 •

edited

Loading

codecov-io commented Dec 12, 2020 •

edited

Loading