Add support for single consumers of pub/sub events

[gctucker]

Currently, all the events are received by all the subscribers when they match the subscription criteria. In some cases, it makes more sense for the event to only be received by one subscriber, for example to distribute jobs across a pool of services.

[JenySadadia]

As per my investigation, Redis Pub/Sub does not provide any way to distribute messages over a group of subscribers.
Redis List or Stream can be helpful in this case. Redis list is a preferable option here if we take into account the complexity of streams.
Investigating it further.

[gctucker]

When publishing a message to a redis queue, only one subscriber will get it. The current implementation creates a queue for each subscriber, so all the messages are sent to all the subscribers. The feature we're missing is to use a single queue with multiple subscribers so that only one of the group of subscribers receives each message.

[gctucker]

Looking at the Redis commands I see there are different ways to dispatch messages. The approach we're using now relies on PUBLISH and SUBSCRIBE, where every message sent with PUBLISH on a channel is received by all the subscribers. This is basically the standard pub/sub approach and the publisher doesn't need to know about subscribers (basically a "broadcast" system).

Another workflow is to use RPUSH for sending messages to a queue and BLPOP to wait for a message on the receiving end. In this scenario, if several clients are running BLPOP only one will receive the message. That works well for load balancing where we want only one client to deal with the event (for example, scheduler services).

With these features, I think we can address all the use-cases with standard Redis functionality. How does that sound?

[gctucker]

When publishing a message to a redis queue, only one subscriber will get it. The current implementation creates a queue for each subscriber, so all the messages are sent to all the subscribers. The feature we're missing is to use a single queue with multiple subscribers so that only one of the group of subscribers receives each message.

Edit: This wasn't really accurate. As explained in the last comment, lists actually work like this (one recipient) but channels despatch messages to all the subscribers.

[JenySadadia]

Looking at the Redis commands I see there are different ways to dispatch messages. The approach we're using now relies on PUBLISH and SUBSCRIBE, where every message sent with PUBLISH on a channel is received by all the subscribers. This is basically the standard pub/sub approach and the publisher doesn't need to know about subscribers (basically a "broadcast" system).

Another workflow is to use RPUSH for sending messages to a queue and BLPOP to wait for a message on the receiving end. In this scenario, if several clients are running BLPOP only one will receive the message. That works well for load balancing where we want only one client to deal with the event (for example, scheduler services).

With these features, I think we can address all the use-cases with standard Redis functionality. How does that sound?

Yes, Redis List can be used here. We can have a publisher that notifies all the subscribers when the queue is updated. Only one consumer will be able to read the message.
Also, I was looking into Redis Streams. It has implemented a consumer group mechanism called XGROUP. The stream distributes messages among the consumers from a group automatically for load balancing.
But for that, we need to completely rework the current Pub/Sub implementation in API.

[gctucker]

But for that, we need to completely rework the current Pub/Sub implementation in API.

Yes, and I think this is fine. We can have a small experiment first to see how this would work. The API endpoints for regular pub/sub with broadcast messages shouldn't need to change from a client point of view, and even if that needed to change we could also do that and now is a good time for this kind of changes before we reach the production-ready stage.

[JenySadadia]

But for that, we need to completely rework the current Pub/Sub implementation in API.

Yes, and I think this is fine. We can have a small experiment first to see how this would work. The API endpoints for regular pub/sub with broadcast messages shouldn't need to change from a client point of view, and even if that needed to change we could also do that and now is a good time for this kind of changes before we reach the production-ready stage.

Makes sense. Yes, we can have a PoC first.

[gctucker]

Redis Streams: https://redis.io/docs/data-types/streams/
Redis Lists: https://redis.io/docs/data-types/lists/

As discussed on Slack, it looks like streams are more for monitoring live events in real-time (e.g. progress of a job on a web dashboard in our case, we we could expose via a websocket) whereas lists are more about inter-processor communication for synchronising work which is what we need for load-balancing. So I believe Redis lists provide what we need for this particular issue.

[JenySadadia]

I have done an initial investigation on this issue with Redis List.

Below is the implementation details of the integration:

• Dropped aioredis package and used redis directly to address the issue mentioned on Replace aioredis package with redis>=4.2.0rc1 #355
• Integrated redis.rpush to push messages on the list with PubSub.publish handler
• Integrated redis.blpop for blocking pop operations with PubSub.listen handler
• Current PubSub mechanism is as-is to keep track of subscriptions and channels. We'll use redis list to push and pop messages instead of broadcasting messages with redis.pubsub whenever new publishers and subscribers are added
• Updated API bindings in the core to accommodate parsing of Redis List messages

Tested the integration with kci pubsub commands.
It works well in terms of load balancing for multiple subscribers with the same subscription ID.

I am working on further testing with pipeline. I'll send a PR with PoC soon.

[gctucker]

Great, thanks for the initial investigation.

Current PubSub mechanism is as-is to keep track of subscriptions and channels. We'll use redis list to push and pop messages instead of broadcasting messages with redis.pubsub whenever new publishers and subscribers are added

I'm not sure what you mean here, we need to support both use-cases so probably different endpoints or at least arguments to subscribe either to broadcast messages or unicast within a pool of subscribers. I believe a separate pair of endpoints would be clearer.

[JenySadadia]

Please see #365