Consider using gRPC as an externally exposed API

[zaa]

grpc (http://grpc.io) has ready-made clients for Java, C++, Go, Python, etc. So Yahoo Pulsar clients would not need to reimplement efficient clients in all the languages (currently exposed websocket interface does not support all the methods provided by the protobuf based protocol, has lower performance and requires creation of a separate websocket connection per topic publisher/consumer).

[agarman]

Are you suggesting gRPC service that uses the C++ lib as an alternative to the Web Sockets service?
Or are you suggesting a rewrite of Java & C++ libs to use gRPC instead of current protobuf based protocol?

[merlimat]

When we started the project gRPC was not available yet so we went with custom protocol.

I think it would be great to offer a gRPC based interface for better integration, though I would see that as an additional layer, such as the WebSocket proxy (which can run embedded in the broker or as a separate component).

One of the primary goals of the custom binary protocol we came up with, was to have the client establish a "session" (either producer/consumer), attached to a topic, perform authentication and then let it publish/receive messages as fast as possible, with flow control to guard the rail.

Eg: we don't want to perform auth at every message, or to specify the topic name each time (which in many cases can be as long as the data itself).

So, mixing the "session" with RPC seems a bit complicated. Also guaranteeing ordering would be challenging as there would be no relation for different publish requests on the same topic.

Having said that, I'd be really happy to have a gRPC based proxy service. Contributions welcome! 😄

It may be also interesting to offer the same interface (or at least a significant portion of it) as GCP pub-sub: https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1

[lzaugg]

Having gRPC as an alternative to the Web Socket service would be awesome. gRPC has built-in support for bidirectional streams - so basically it could be seen as a session (it's using HTTP/2 streams), no? Guaranteeing the message order shouldn't be a problem then. I think it's also important to know if pulsar is going to loosen the message ordering constraints (at least for such an "additional" layer) because it would make things easier (for use cases where ordering is not important - in the same way as GCP pub/sub doesn't guarantee any order).

[sijie]

Having gRPC as an alternative to the Web Socket service would be awesome. gRPC has built-in support for bidirectional streams - so basically it could be seen as a session (it's using HTTP/2 streams), no?

agreed. I believe ordering is not a problem with gRPC bidirectional streaming. It is actually very easier and super fun on using gRPC bidirectional streaming.

I think the most interesting piece here is to add a GCP pub/sub proxy with gRPC protocol.

I think it's also important to know if pulsar is going to loosen the message ordering constraints (at least for such an "additional" layer) because it would make things easier

in the context of "shared" subscription, the message ordering constraints are already relaxed. that said you can use exclusive/failover subscription for ordered message consumption, shared subscription for non-ordered message consumption.

[RobIsHere]

Whole Clusters run on grpc like k8s + istio and it's well supported by proxies like envoy.

You could think about clients talking to their user's topics directly, authenticated by ingress - e.g. envoy filters (see https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/grpc_web_filter for ideas).

IMHO, I'm not convinced about GCP pub-sub. Making one thing like the other is almost often a large effort and a bad fit if you look into details. When google changes the api, do you follow?
Wrapping your already proofen, tested and implemented custom protocol in grpc is probably a huge time saver. And your clients apis are well-designed like they are. Better early than google like ;)

[snoodleboot]

I would love to see this! I can understand about pub/sub. It really doesn't serve the same purpose as Pulsar or any distributed log. Different use cases.

[cbornet]

+1. Supporting gRPC would give access to a lot more clients, integration with reactive frameworks (like RxJava or Reactor), provide application-level flow control, etc...
Another stream protocol to watch IMO is RSocket.io which has built-in integration of reactive-streams spec (non blocking streams with back-pressure). Since it was just released, it lacks client SDKs but that should evolve over time.

[cbornet]

One of the primary goals of the custom binary protocol we came up with, was to have the client establish a "session" (either producer/consumer), attached to a topic, perform authentication and then let it publish/receive messages as fast as possible, with flow control to guard the rail.
Eg: we don't want to perform auth at every message, or to specify the topic name each time (which in many cases can be as long as the data itself).
So, mixing the "session" with RPC seems a bit complicated. Also guaranteeing ordering would be challenging as there would be no relation for different publish requests on the same topic.

gRPC can establish a session for bidirectional streaming ! So IMO it could totally be used as the base protocol for Pulsar. The definition would look something like

service Pulsar {
    rpc exchange(stream BaseCommand) returns (stream BaseCommand);
}

Instead of passing auth info via CommandConnect, you would pass them as gRPC's Metadata fields (similar to HTTP headers).
Note that this reuses the PulsarApi.proto, so a lot of code would be unchanged I think.
You would probably still need drivers because some functionalities require cooperation between the client and the server. But these drivers would be easier to write.
gRPC also has auth mecanisms built-in that could maybe be reused and has built-in flow control.

That said I have started the work on a gRPC proxy. Consumption and production are working. I need to clean it up then I'll do a PR.

[cbornet]

And if I'm not mistaken BookKeeper uses gRPC internally, so it would be coherent to make it the base protocol in Pulsar also.

[sijie]

That said I have started the work on a gRPC proxy. Consumption and production are working. I need to clean it up then I'll do a PR.

Look forward to your PR.

And if I'm not mistaken BookKeeper uses gRPC internally, so it would be coherent to make it the base protocol in Pulsar also.

gRPC is used only used for bookkeeper's table service. but the ledger service is still using custom protocol. but agreed with you, gRPC has very rich ecosystem, it is a good direction to good in general.

[merlimat]

@cbornet The reason we haven't used gRPC is that it wasn't available when we started, so we went with custom protocol over protocol buffer. After that, migrating the internal protocol was a big step.

[cbornet]

Yes. That's a very good reason indeed. Maybe in the future 😄 . I can understand there are bigger priorities.