cpp/python API: please log to stderr instead of stdout by default

[candlerb]

Is your feature request related to a problem? Please describe.

The pulsar python client is a thin wrapper around the C++ API. When it runs, it generates debugging messages, e.g.

2019-11-11 16:01:46.169 INFO  ConnectionPool:72 | Created connection for pulsar://localhost:6650
2019-11-11 16:01:46.170 INFO  ClientConnection:324 | [127.0.0.1:51970 -> 127.0.0.1:6650] Connected to broker
2019-11-11 16:01:46.177 INFO  HandlerBase:52 | [persistent://public/default/syslog, reader-1d1d3c, 0] Getting connection from pool
2019-11-11 16:01:46.178 INFO  ConnectionPool:72 | Created connection for pulsar://ldex-pulsar.int.example.net:6650
2019-11-11 16:01:46.179 INFO  ClientConnection:326 | [127.0.0.1:51974 -> 127.0.0.1:6650] Connected to broker through proxy. Logical broker: pulsar://ldex-pulsar.int.example.net:6650
2019-11-11 16:01:46.183 INFO  ConsumerImpl:170 | [persistent://public/default/syslog, reader-1d1d3c, 0] Created consumer on broker [127.0.0.1:51974 -> 127.0.0.1:6650]

However, these messages are generated on stdout, not stderr. As a result, this causes problems for any client program which wishes to use stdin and stdout for data communication.

Example: an rsyslog omprog program receives messages on stdin and sends acknowledgements on stdout. The debug messages sent by the pulsar API confuse rsyslog and cause it to crash.

Describe the solution you'd like
If debug messages are to be enabled by default, I would like them to be written to stderr rather than stdout.

That is a change of behaviour. If that's considered a bad thing at this stage, then it could be controlled by a flag.

Describe alternatives you've considered
There is an option pulsar.Client(log_conf_file_path=...) which suggests it can be configured.

However, it doesn't appear to do anything. I can set it to the name of a file, and strace does not show that file being opened. I can set it to a non-existent file, and no error occurs.

$ cat testconf.py
import pulsar
client = pulsar.Client('pulsar://localhost:6650', log_conf_file_path="/tmp/log4cpp.conf")
msg_id = pulsar.MessageId.earliest
reader = client.create_reader('my-topic', msg_id)
reader.read_next(100)

$ strace -f python3 testconf.py 2>strace.out
...

$ grep log4cpp strace.out
$

Note: I am using pulsar-client (2.4.1) installed via pip3 under Ubuntu 18.04/python 3.6.8.

Additional context
I see that there are two logger implementations in lib/Log4cxxLogger.cc and lib/SimpleLoggerImpl.cc

It looks like the SimpleLogger is default, in CMakeLists.txt:

option(USE_LOG4CXX "Build with Log4cxx support" OFF)

And as far as I can see, SimpleLoggerImpl is hard-coded to write to stdout:

    void log(Level level, int line, const std::string &message) {
        std::stringstream ss;

        printTimestamp(ss);
        ss << " " << level << " " << _logger << ":" << line << " | " << message << "\n";

        std::cout << ss.str();
        std::cout.flush();
    }

This makes it very hard to redirect the logging. It looks like I might have to fork a child, talk to it on fd 3, and copy data back and forth.

[candlerb]

I have a workaround:

    fd = os.dup(1)    # copy the original stdout
    os.dup2(2, 1)     # join stdout onto stderr
    new_stdout = os.fdopen(fd, "w")  # wrap the original stdout

... then write to new_stdout instead of stdout. Anything which is written to fd 1 gets sent to stderr.

[merlimat]

We could switch the messages to stderr by default. Also, in master there's the option to configure the logger function from Python directly.

[candlerb]

Also, in master there's the option to configure the logger function from Python directly.

It's master I am looking at. Which option are you referring to? All I could find was log_conf_file_path, which apparently doesn't do anything when not built against log4cxx.

[merlimat]

Oh, it was not merged yet: #5279

[dhagrow]

What is the plan for this feature? I see there was a solution (#6113) which was reverted (#6174) because it changed the order of arguments to the client's constructor.

It seems like a simple solution would be to add the logger argument to the end of the constructor. Or, if not, at least allow the logger to be set on the client object.