we should add monitor about whether the netty worker thread pool are busy or not

[kkewwei]

We find the took in response is smaller than the external statistics timeout in product, part of the reason is that some threads of worker thread pool are relatively busy, the search requests are pending on the worker queue. Yet there is not available api, to get the statistics of worker queue, we have to add metrics and build the source code, If fact, There is a serious queuing here:
97, 64, 68, 108, 133, 61, 62, 123, 70, 66, 118, 110, 65, 52, 95, each number represents one of the worker threads's pending tasks.

If we can add api to get those metrics.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[kkewwei]

hi, @romseygeek, can you please confirm it? Too much timeout are used in ES, some request is very important(such as leader check and follower check), if in the case, It's very difficult to troubleshooting.

[kkewwei]

In our product, the count of the worker threads's pending tasks in the data node is: 793, 387, 852, 43, 988, 119, 115, 809, 761, 46, 47, which causes the master to detect the data node timeout, and the data node will be kicked out of the cluster by the master,

The removed operation happened in the early morning, since there was no indicator before, the kicking operation was very strange, If we can add api to get those metrics in es.

Of cause I will continue to check why there are so many tasks in the work queue.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

What version are you using @kkewwei and how are you counting the number of tasks pending on each worker thread?