Plugin to pipe stdout/stderr to a log event

[mrocklin]

Folks sometimes want stdout and stderr sent to the client. This is maybe pretty easy with ...

1. Teeing off sys.stdout/stderr as in https://stackoverflow.com/questions/616645/how-to-duplicate-sys-stdout-to-a-log-file
2. Dropping the results into the print log event
3. All withing a WorkerPlugin

Maybe something like this for stdout?

class Tee(object):
    def __init__(self, worker):
        self.stdout = sys.stdout
        self.worker = worker
        sys.stdout = self
    def close(self):
        sys.stdout = self.stdout
    def write(self, data):
        self.worker.log_event("print", data)
        self.stdout.write(data)
    def flush(self):
        self.file.flush()

class StdOut(WorkerPlugin):
    def start(self, worker):
        self.tee = Tee(worker)

    def close(self):
        self.tee.close()

[fjetter]

I think this is a duplicate of #2033

[maxbane]

This would be a nice follow-up to the recently revamped distributed.print() function, which is a drop-in replacement for the built-in print that task authors can use to selectively forward their print output from worker to client. Redirecting stdout/stderr as proposed here would be for cases where you want everything forwarded, regardless of whether you control the originating print/write calls. I suppose users should be aware that this could put a lot of performance pressure on the scheduler if there are a lot of writes to forward from a lot of workers.

I think it is a little different than #2033, which is asking to have stdout/stderr forwarded into a client-side python logger, rather than just forwarding to client-side stdout/stderr. I like the simplicity of simply forwarding to stdout/stderr, but if PR #7276 is merged, it would be pretty straightforward to forward to a logger, too, by re-using the "forwarded-log-record" topic introduced by that PR.