TiDB is unable to handle a network partitioning

[den-crane]

I do this weird test to reproduce a real production issue, sometimes it happens in our network.

I have a simple setup with 5 hosts, 5 of everything
Fresh/empty installation, no any additional configuration.

host1 # tiup cluster display tidb1
Cluster type:       tidb
Cluster name:       tidb1
Cluster version:    v8.5.3
Deploy user:        tidb
SSH type:           builtin
Dashboard URL:      http://host2:2379/dashboard
Dashboard URLs:     http://host2:2379/dashboard
ID           Role     Host   Ports                            OS/Arch       Status   Data Dir                 Deploy Dir
--           ----     ----   -----                            -------       ------   --------                 ----------
host1:2379   pd       host1  2379/2380                        linux/x86_64  Up       /tidb-data/pd-2379       /tidb-deploy/pd-2379
host2:2379   pd       host2  2379/2380                        linux/x86_64  Up|L|UI  /tidb-data/pd-2379       /tidb-deploy/pd-2379
host3:2379   pd       host3  2379/2380                        linux/x86_64  Up       /tidb-data/pd-2379       /tidb-deploy/pd-2379
host4:2379   pd       host4  2379/2380                        linux/x86_64  Up       /tidb-data/pd-2379       /tidb-deploy/pd-2379
host5:2379   pd       host5  2379/2380                        linux/x86_64  Up       /tidb-data/pd-2379       /tidb-deploy/pd-2379
host1:4000   tidb     host1  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host2:4000   tidb     host2  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host3:4000   tidb     host3  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host4:4000   tidb     host4  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host5:4000   tidb     host5  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host1:9000   tiflash  host1  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host2:9000   tiflash  host2  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host3:9000   tiflash  host3  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host4:9000   tiflash  host4  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host5:9000   tiflash  host5  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host1:20160  tikv     host1  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160
host2:20160  tikv     host2  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160
host3:20160  tikv     host3  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160
host4:20160  tikv     host4  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160
host5:20160  tikv     host5  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160

Everything is great.
But if I break the network between two hosts, for example between host1 and host3, then TiDB became readonly till the network is fixed.

Let's break the network between two hosts:

host1 #  ip route add host3 reject  // it can be package drops (blackhole), does not matter, in the production ping shows timeouts

host1 # ping host3
ping: connect: No route to host

host1 # tiup cluster display tidb1
Cluster type:       tidb
Cluster name:       tidb1
Cluster version:    v8.5.3
Deploy user:        tidb
SSH type:           builtin
Dashboard URL:      http://host2:2379/dashboard
Dashboard URLs:     http://host2:2379/dashboard
ID           Role     Host   Ports                            OS/Arch       Status   Data Dir                 Deploy Dir
--           ----     ----   -----                            -------       ------   --------                 ----------
host1:2379   pd       host1  2379/2380                        linux/x86_64  Up       /tidb-data/pd-2379       /tidb-deploy/pd-2379
host2:2379   pd       host2  2379/2380                        linux/x86_64  Up|L|UI  /tidb-data/pd-2379       /tidb-deploy/pd-2379
host3:2379   pd       host3  2379/2380                        linux/x86_64  Down     /tidb-data/pd-2379       /tidb-deploy/pd-2379
host4:2379   pd       host4  2379/2380                        linux/x86_64  Up       /tidb-data/pd-2379       /tidb-deploy/pd-2379
host5:2379   pd       host5  2379/2380                        linux/x86_64  Up       /tidb-data/pd-2379       /tidb-deploy/pd-2379
host1:4000   tidb     host1  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host2:4000   tidb     host2  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host3:4000   tidb     host3  4000/10080                       linux/x86_64  Down     -                        /tidb-deploy/tidb-4000
host4:4000   tidb     host4  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host5:4000   tidb     host5  4000/10080                       linux/x86_64  Up       -                        /tidb-deploy/tidb-4000
host1:9000   tiflash  host1  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host2:9000   tiflash  host2  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host3:9000   tiflash  host3  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host4:9000   tiflash  host4  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host5:9000   tiflash  host5  9000/3930/20170/20292/8234/8123  linux/x86_64  Up       /tidb-data/tiflash-9000  /tidb-deploy/tiflash-9000
host1:20160  tikv     host1  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160
host2:20160  tikv     host2  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160
host3:20160  tikv     host3  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160
host4:20160  tikv     host4  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160
host5:20160  tikv     host5  20160/20180                      linux/x86_64  Up       /tidb-data/tikv-20160    /tidb-deploy/tikv-20160

From user's perspective TiDB became readonly:

MySQL [test]> drop table test;
ERROR 8027 (HY000): Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV

MySQL [test]> insert into test values(1);
ERROR 9005 (HY000): Region is unavailable

Any clues how to deal with it?
Do I need more nodes?

[den-crane]

Thought I realized that its because I do queries at a host1.

At host2 it's not readonly (DML work):

insert into test values(1);
Query OK, 1 row affected (0.009 sec)

except DDL hangs forever of fail.

create table test2 ( A int) ;
ERROR 9005 (HY000): Region is unavailable