[STORM-3514] "topology.enable.message.timeouts: false" has no effect on ackers

[jira-importer]

"topology.enable.message.timeouts: false" does prevent tuples from being failed if not acked in "topology.message.timeout.secs" seconds, but it still prevents __ackers from acking anchored tuples to the spout. When used with "topology.max.spout.pending" this effectively stalls the spout completely as the tuple is neither failed, nor acked.

---

Originally reported by mevgheni, imported from: "topology.enable.message.timeouts: false" has no effect on ackers

• status: Closed
• priority: Major
• resolution: Won't Fix
• resolved: 2024-07-02T09:47:00+00:00
• imported: 2025-01-24

[jira-importer]

srdo:

I think you should reconsider whether you want to use that configuration. When you disable message timeouts, any unacked tuple will stay in the spout's pending set forever. Once you hit the max spout pending limit, the spout will stall permanently. As tuples may fail to be acked due to both coding errors and runtime network issues (e.g. a lost message from an acker), this will likely happen at some point during your topology's lifetime. I'm wondering if this is what you're seeing?

Disabling message timeouts can be useful if you've also disabled acking, as you then save a bit of processing, but if you have acking enabled you probably don't want to disable timeouts.

[jira-importer]

mevgheni:

I use acking mostly for throttling the spout, as whole-topology propagation varries from fractions of milliseconds to multiple minutes, yet spout produces them at millions per second if left unthrottled. Most messaging is inner-worker, so network related issues shouldn't be a problem. Failing of tuples on expected errors is done manually, besides that, the topology is intentionally fail-fast.

 

What I'm seeing is a complete stall of a spout withing 1st minute of running. Looking at emited touples, I see that initially everything works fine until the propagation exceedes the "topology.message.timeout.secs" when __acker stops acking to the spout, even thought bolts keep executing and acking to the __acker. What I'm left with is a tuple, that has been processed and acked in the bolt, but not acked to the spout. As I have timeouts disabled, the spout does not reemit (which is not even needed, as the touple was already processed and acked), but as the spout didnt receive the ack from the __acker, it keeps waiting for queue to free up (which already happened, as the touple was processed and acked by the bolt). Setparately setting timeout.secs to a non-default and much higher value lets the topology run as it should. This all tells me that there is a bug in the __acker code that prevents it from acking touples that exceed "topology.message.timeout.secs" even when "topology.enable.message.timeouts" is set to false.

[jira-importer]

srdo:

I took a look at where Storm actually uses the "topology.message.timeout.secs" parameter, and it is only used here https://github.com/apache/storm/blob/v1.2.3/storm-core/src/clj/org/apache/storm/daemon/executor.clj#L361

What happens in that line is when we decide whether or not to enable tick tuples for an executor. Tick tuples are sent periodically to bolts for timekeeping, and are used for timing out tuples in the spout, and to make sure that ackers don't keep around data for failed tuples for longer than necessary. Tick tuples can only affect acking of tuples in the sense that they can cause the ackers to throw away data about still live tuples. As you may note in the linked code, ackers still receive ticks even when "topology.message.timeout.secs" is false. This may or may not have been intentional, but it's probably why you're seeing this issue. When the normal tuple timeout (30s) is reached, the acker will throw away data about your live tuple, so when it receives acks it doesn't know what to do with them.

We can fix this by also disabling tick tuples for ackers if the "topology.message.timeout.secs" is false.

As an aside, you may consider upgrading to Storm 2.0.0, as it has a new backpressure mechanism that doesn't depend on the ackers, so you could probably disable acking entirely if you upgraded to that version.

[jira-importer]

rzo1:

Storm 1.x isn't maintained anymore. If the issue still applies for 2.6.x, feel free to re-open.