IPv6 Support for broker and proxy with bindAddress

[0asys]

Describe the bug
the default setting for binding in broker.conf and proxy.conf is bindAddress=0.0.0.0.

pulsar/conf/proxy.conf

Line 53 in 3e06571

bindAddress=0.0.0.0

the specified default 0.0.0.0 binds to ipv4 only, so the service is not reachable over ipv6.

[root@proxy pulsar]# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 0.0.0.0:6650            0.0.0.0:*               LISTEN      41186      8351118    2470721/java
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      41186      8349589    2470721/java

this setting does not accept a a ipv6 address [::0], ::0 or a ipv6 address with or without brackets.

available error in logs:

2022-03-17T10:52:55,986+0100 [sun.misc.Launcher$AppClassLoader@1d56ce6a] error Uncaught exception in thread main: Failed to bind Pulsar Proxy on port 6650

To Reproduce
Steps to reproduce the behavior:

1. Set a ipv6 address in broker.conf or proxy.conf for bindAdress: bindAddress=[::0]
2. restart pulsar broker or proxy
3. check listening ports - no pulsar listen on ::6650

Expected behavior
pulsar broker and proxy listens on ipv6 address.

Desktop (please complete the following information):

• OS: RedHat 8
• Pulsar v2.9.1

Additional context

• related issue: pulsar-standalone: use ipv6 error  #8231
• There was a related merge request for the "advertised service urls", which claims to resolve the issue, but its only resolved for service urls, not the binding/listening: IPV6 support not available #954

[eolivelli]

are you able to troubleshot why this is happening ?
like putting some debug

[0asys]

unfortunately, setting log levels to debug with env vars:

PULSAR_LOG_LEVEL=debug
PULSAR_LOG_ROOT_LEVEL=debug

does show debug logs, but no additional errors or hints for this specific problem.

is jetty used for this?
the default bind address is set to 0.0.0.0 in the codebase:

pulsar/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java

Line 227 in 342a5df

private String bindAddress = "0.0.0.0";

and there is a jetty discussion, if the bind address is not defined, both stacks (4 & 6) are considered by default:
https://www.eclipse.org/lists/jetty-users/msg09643.html

[eolivelli]

we are using Jetty for the REST API part and Netty for the Binary part

[0asys]

error originating from:

pulsar/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyService.java

Line 210 in 3a337b8

throw new IOException("Failed to bind Pulsar Proxy on port " + proxyConfig.getServicePort().get(), e);

I think this is the relevant part for binding in netty:
https://github.com/netty/netty/blob/d34212439068091bcec29a8fad4df82f0a82c638/transport/src/main/java/io/netty/bootstrap/AbstractBootstrap.java#L252

references here for the properties, where lobmok is used:

pulsar/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConfiguration.java

Line 143 in 3a337b8

private String bindAddress = "0.0.0.0";

convert with SocketUtils.socketAddress string to InetSocketAddress:
https://github.com/netty/netty/blob/40382fb1af275e931a765c5001f4b6bd8061991e/common/src/main/java/io/netty/util/internal/SocketUtils.java#L177

but i don't see the thrown exception somewhere in the output or log, but there should be one from the thrown IOException

[eolivelli]

if I remove all the occurrences of "-Djava.net.preferIPv4Stack=true" inside "bin" (or you can set it to "false") then I am able to boot Pulsar standalone and the proxy on my Mac with JDK11

[0asys]

jup, this works.

set bindAddress=0.0.0.0 in proxy.conf and removing -Djava.net.preferIPv4Stack=true in bin/pulsar is indeed working.

why is the setting java.net.preferIPv4Stack even in place? Should this not be as the default (false) and enabling dualstack?
IMHO this should be standard today.

[eolivelli]

I believe that it is only because support for IPV6 is not fully available in many libraries (like parsing addresses..) and no one dares to enable it.

If you are able to successfully run all Pulsar components without issues, then please advertise on user@pulsar or dev@pulsar and we can remove that flag, or make it configurable somehow.

[wangjialing218]

I have already test IPV6 connections between pulasr client and broker, as long as kafka client and kop, with -Djava.net.preferIPv4Stack=false, there seems no problem.
And I'm going to have a IPV6 test between broker and bookie and zookeeper, after than we could make it default false.