automatic batching interface for cloud logging client

[faithseed]

The logging API support batching mode manually, where you can put multiple entry in the log.write()

log.write([entry, secondEntry]);

Is that possible to implement an batching interface to support this automatically?

for example:

log.add(entry);
...
log.add(another_entry);

In the background, logging client does periodic flush internally sending all buffered logs to cloud logging automatically. The flush is triggered when enough entries are buffered and/or at a fixed interval.

This would make the API better performance wise by making less APIs calls to cloud logging?

[stephenplusplus]

I think this is better resolved in the application layer and not our library, since every use case would likely have custom requirements around what action triggers a flush.

@filipjs do you think this is something we should try to support?

Also to consider, we use gRPC for our Logging API, which maintains an open connection to send multiple requests through. That eliminates the overhead of making several HTTP/1 calls, so it's possible the performance wins could be negligible.

[faithseed]

Thanks @stephenplusplus.

What is performance difference between

log.write([entry, secondEntry]);

and

log.write([entry]);
log.write([secondEntry]);

If they are considered equally efficient, maybe we should document that in the developer doc and encourage users to choose either one of them.

[stephenplusplus]

We haven't measured the performance difference, and I think it might be micro-optimization if we did or recommended an approach one way or the other. I think it's best left to the application and its concerns to make the best decision for their use.

[filipjs]

Our documentation recommends batching as it's more efficient and doesn't consume QPS quota that quickly (https://cloud.google.com/logging/quota-policy)
I'm not sure how gRPC is counted for QPS purposes.

[stephenplusplus]

@jgeewax any insight into how QPS are measured when using gRPC?

[jgeewax]

@rimey : Any idea? Basically is "1 write log entry" equivalent to a single gRPC call? or is it the number of log entries written in the gRPC call?

Or in more specific terms, is log.write([entry1, entry2]) counted as 1 log write entry? or 2?

[stephenplusplus]

Friendly ping @rimey -- or anyone else who can help.

cc @ofrobots

[ofrobots]

This thread conflates performance implications of batching with quota implications.

Or in more specific terms, is log.write([entry1, entry2]) counted as 1 log write entry? or 2?

I am pretty sure that this is a single call. I think the more interesting question is this:

log.write(foo);
log.write(bar);

Are those counted as two entries.write calls? If so, as I expect, we need buffering.

[zbjornson]

I'm 99% sure the protocol doesn't matter; it's the actual command that counts. (bunyan-stackdriver is built on google-cloud/logging and thus uses gRPC, and I know I hit the quota, although that was ages ago.)

From the docs:

To improve throughput and to avoid exceeding the quota limit for calls to entries.write, you should write multiple log entries at once rather than calling this method for each individual log entry.

The approach in bunyan-stackdriver is to flush the buffer every 500 ms (default; configurable). I have yet to find a maximum total payload size, only a per-Entry size (112640 bytes), so there's no forced flush if the buffer gets to be a certain size (although that would be good for controlling memory usage). cloud-trace-nodejs does both time batching and buffer size control.