Offering my help to build stuff

[darioalessandro]

Hey mate, this project looks great.

I am building a video streaming platform: videocall-rs https://github.com/security-union/videocall-rs

We currently use NATs, it works great but I am very interested in contributing to a pure rust message streaming platform that is more aligned with our stack and values.

What really called my attention is that you are planning to add a REST api, with NATs we all end up just writing this by hand, so this really makes me happy

How can we help you? what are you working on?

CC @griffobeid

[spetz]

Hey Dario,

Thanks for the kind words :) Actually, I've been following your YT channel for a while now, and have seen the recent news about the video conferencing project, really cool stuff!

Regarding the REST API, it's been implemented some time ago already (based on Axum), as one of the 3 available transports (raw TCP, QUIC and HTTP).

The endpoints are listed here (it does use VS Code REST Client extension), and since all the transport protocols implement the same Client trait, you can already make use of the underlying HttpClient implementation.

Of course, it's not as performant as raw TCP + it's a stateless connection, thus the features such as consumer group aren't supported, but still, it works quite well. You can play on your own with the existing benchmarking app or sample consumer + producer app, to try out the different scenarios using the different protocols etc. (BTW in the getting started docs, you can simply replace the TcpClient with HttpClient to try out HTTP API).

I thought of adding WebSocket and/or WebTransport protocol at some point in the future (the plain QUIC is already there), so I guess this one could be a better fit for the video streaming platform? Moreover, since the Iggy server is a persistent stream (like Kafka or RabbitMQ Streams), does it make sense for you to persist the data in the form of an append-only log, or maybe the potential feature of being just some kind of in-memory buffer passing the messages through would make more sense?

I've been planning to start working on the cluster + replication in the upcoming weeks (this will be a huge and complex topic), but I'm open to other ideas and features in the meantime. Besides that, I would also like to put even more emphasize on the overall performance optimizations, stuff like zero-copy directly from the disk/memory to the network buffer, experimenting with io_uring and so on.

Feel free to ping me on LinkedIn or Twitter, we could even set up a call and discuss the ideas :)

[darioalessandro]

@spetz thanks for your response mate!!

Thanks for the pointers about how to use the rest API, I am looking at this right now!!

I thought of adding WebSocket and/or WebTransport protocol at some point in the future (the plain QUIC is already there), so I guess this one could be a better fit for the video streaming platform? Moreover, since the Iggy server is a persistent stream (like Kafka or RabbitMQ Streams), does it make sense for you to persist the data in the form of an append-only log, or maybe the potential feature of being just some kind of in-memory buffer passing the messages through would make more sense?

Regarding adding Webtransport/WebSocket I think this will open the door to connecting web clients!!!

Do you support different QOS? or is it all QOS 2, per MQTT definition:
QOS 0: Delivery is not guaranteed
QOS 1: Delivery is guaranteed at least once
QOS 2: Delivery is guaranteed only once

I've been planning to start working on the cluster + replication in the upcoming weeks (this will be a huge and complex topic), but I'm open to other ideas and features in the meantime. Besides that, I would also like to put even more emphasize on the overall performance optimizations, stuff like zero-copy directly from the disk/memory to the network buffer, experimenting with io_uring and so on.

this is huge!! you have a chance to disrupt kafka!! it took them years to ditch zookeeper!! you get the benefit of starting fresh!!

Yes, I would love to bounce some ideas, look at how NATs, rabbitMQ does clustering and finding ways to innovate in this area!

[spetz]

I'll think of doing some research in the WebSocket area in the upcoming weeks then, Axum seems to be supporting them, so it shouldn't be too difficult I guess.

Regarding QOS, since this is a message a stream, not a regular message broker, the messages are never lost as they belong to the append-only log (unless there's a retention policy specified, which hasn't been implemented yet). Just like in Kafka or in RabbitMQ Streams, it's on the client to poll the messages based on the provided offset. You can easily achieve e.g. at-most-once delivery with the auto commit enabled (storing the so-called consumer offset on the server side), or at-least-once delivery, given that the consumer keeps track of the offset on its own and can deal with the potential message duplication (e.g. the app has crashed after processing the message and consumer didn't store the latest offset locally) - these are the typical challenges when working with either message brokers or streaming :)

From the message producer perspective (or actually based on the server configuration), you can control the consistency guarantees with the settings such as messages_required_to_save, enforce_sync or an additional message_saver component running in the background. The client has no impact on the pending messages. If the server crashes (it's not a graceful shutdown) and you have disabled message_saver background task or the interval is too long, it's possible that some buffered messages could be lost, but again - it's up to you to find the best trade-off between the overall throughput, performance and consistency :)

In the case of the horizontal scaling, it does support consumer groups (similar to Kafka) and sending the messages to the partition ID being calculated on the server side based on the provided key using murmur3 hash (take a look at the getting started guide where I describe SendMessages command). Also, in the configuration section, I added a bit more info about the mentioned settings.

Speaking of the clustering, I'm aware that this is a huge and complex topic, and I'll be more than happy to discuss the possible ideas. I might end up using Raft or some other, well-established and mature consensus algorithm, but no decisions have been made so far.

[hubcio]

Closing this, to be further discussed on discord.