Broadcast/per-silo grains

[ReubenBond]

In #587, we discussed the occasional desire to break location transparency of Virtual Actors and have the ability to message a grain (or otherwise) on an individual silo to broadcast a message to all silos.

In my case, I would have been happy to kick off a stream consumer inside a bootstrap provider and push messages to it using a stream, but this is not currently supported.

Use cases include:

• Configuration
• Cache management
• Metrics collection
• Fault injection
• Process control (eg, live code reload)
• Implementing certain distributed algorithms, eg Raft & Paxos

So, how should we do this?

[gabikliot]

@ReubenBond , is it only for "broadcast a message to all silos"? So from an API perspective you have a way to identify a group of grains and sending ONE msg to the whole group of grains and don't know how much are in the group and don't get to message each one individually?
This is a bit different from "break location transparency of Virtual Actors".

[ReubenBond]

@gabikliot for most of the use cases, yes. Although that would probably not be the nicest way to perform metrics collection. @jthelin likely has some ideas.

[gabikliot]

@ReubenBond , why not use the standard solution we recommend for this case:
Bootstrap provider makes a call into a PreferLcalPlacement grain with random id and that grain will subscribe to the stream?

[jthelin]

The other part of the "Per Silo" grain semantics is that those grain instances should never "roam" off their designated silo.

If that silo crashes, that grain is just gone, and would never be recreated on another silo.

A (simplified) concrete scenario example is if you have a "shard" of data physically located on one machine, then the grain responsible for dealing with that shard can only run on that one physical machine.

The current PreferLocalPLacement policy handles the initial "local" placement part, but not the "guaranteed no-roam" part.

I think my scenario is a slightly more specific and constrained version of the general requirement that @ReubenBond is describing above.

[gabikliot]

Do you guys have concrete ideas how the new API and semantics (w.r.t. failures) for this new type of grain might look like?

[jthelin]

From my perspective, the failure mode can be very simple:

If silo is gone, then the per-silo grain(s) in that silo are also gone.

I would ideally prefer a specific exception with well understood semantics (maybe something like GrainNoLongerActiveException ?) and [preferably] fast-track throw (as soon as the cluster membership change is detected) rather than needing to wait full timeout period for an eventual generic TimeoutException exception.
That makes it easier for any "coordinator" functions in my app to make better informed decisions about shard availability, rather than trying to second guessing whether timeout is from temporary communications glitch [retriable] or because silo just crashed / went offline [dead].

When new silo restarts on a failed host (i.e. same host / ip address but different silo generation number) then Orleans runtime will run the usual app boostrap provider plugins in that silo, which would recreate a fresh instance of the per-silo grain(s) on that silo.

[ReubenBond]

I am on-board with @jthelin's suggestion. It still requires us to add some application-specific logic in the bootstrap provider, though.

We could create a [SiloSingleton] marker attribute and automatically activate each marked grain type during bootstrap.

[jthelin]

BTW, the new [StatelessWorker(1)] functionality in PR #688 should now provide the "no-roam" guarantee i was talking about above, rather than using [PreferLocalPLacement] which is only best-efforts.

Updated: This approach does not work, for subtle reasons that @gabikliot pointed out to me in e-mail and hinted at below. :(

[veikkoeeva]

@jthelin An excellent point! Would it a worthy point to document in FAQ, for instance?

[jthelin]

I am hoping to upload a code sample this weekend, but I will look at the docs too ;)

[gabikliot]

Just to keep in mind, this is still StatelessWorker, which means you can have more than one globally. There will be up to 1 on every silo, but each silo can have 1 and globally there may be many.

Also, the above means you can not address an individual activation of the stateless worker globally. There is no way (currently, in the current implementation, unless we change the programming model) to send a message from silo X to StatelessWorker on silo Y. You can address local activation of the StatelessWorker. So for example, every BootstrapProvider will be talking to its own, single copy, of the StatelessWorker(1) grain.