Enable TCP/IP keepalive for all ZK client connections in all components started with bin/pulsar

[lhotari]

Motivation

This is a follow-up on #12982 which enabled TCP/IP keepalive for ZK server side.
This PR enables TCP/IP keepalive for all ZK client connections in all components

• Enable TCP keepAlive flag on the sockets used for client connections
  • see https://issues.apache.org/jira/browse/ZOOKEEPER-3712 for details
    • "Some broken network infrastructure may lose the FIN packet that is sent from closing client.
      These never closed client sockets cause OS resource leak. Enabling this option terminates
      these zombie sockets by idle check. ... Currently this option is only available
      when default NIOServerCnxnFactory is used."

Modifications

add -Dzookeeper.clientTcpKeepAlive=true to OPTS in bin/pulsar script so that the setting is passed to all components that are started using bin/pulsar script.

[github-actions]

@lhotari:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

[github-actions]

@lhotari:Thanks for providing doc info!

[Jason918]

LGTM

[michaeljmarshall]

LGTM

[lhotari]

It turns out that ZooKeeper TCP keepalive settings contain a major gap.

docs: https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#:~:text=tcpKeepAlive%20%3A%20(Java%20system,NIOServerCnxnFactory%20is%20used.

tcpKeepAlive : (Java system property: zookeeper.tcpKeepAlive) New in 3.5.4: Setting this to true sets the TCP keepAlive flag on the sockets used by quorum members to perform elections. This will allow for connections between quorum members to remain up when there is network infrastructure that may otherwise break them. Some NATs and firewalls may terminate or lose state for long-running or idle connections. Enabling this option relies on OS level settings to work properly, check your operating system's options regarding TCP keepalive for more information. Defaults to false.

clientTcpKeepAlive : (Java system property: zookeeper.clientTcpKeepAlive) New in 3.6.1: Setting this to true sets the TCP keepAlive flag on the client sockets. Some broken network infrastructure may lose the FIN packet that is sent from closing client. These never closed client sockets cause OS resource leak. Enabling this option terminates these zombie sockets by idle check. Enabling this option relies on OS level settings to work properly, check your operating system's options regarding TCP keepalive for more information. Defaults to false. Please note the distinction between it and tcpKeepAlive. It is applied for the client sockets while tcpKeepAlive is for the sockets used by quorum members. Currently this option is only available when default NIOServerCnxnFactory is used.

The part "Please note the distinction between it and tcpKeepAlive. It is applied for the client sockets while tcpKeepAlive is for the sockets used by quorum members. Currently this option is only available when default NIOServerCnxnFactory is used."

• the clientTcpKeepAlive is only applied on the server side for client sockets and it only handles NIOServerCnxnFactory
• there is no setting to enable tcp keep-alive on the client side