Scheduler Broadcast Job Type #83
Conversation
Signed-off-by: joshvanl <[email protected]>
| Since the job will be created abstractedly from the application by the runtime, the name will be generated with the following form; `__internal__$actor-type__$actor-id`. | ||
| This job type will return an `already exists` error if a PUT is attempted on an existing durable Actor ID. | ||
| The data field of the job will include a `common.InvokeRequest` proto message, detailing how the actor will be invoked. | ||
| The HTTP Verb of the invocation request defined will be ignored, in favour of always using a PUT or DELETE. |
There was a problem hiding this comment.
Minor nit: I would prefer POST just because PUT implies an update, but POST almost always implies creating something new.
There was a problem hiding this comment.
Thanks- this is a mistake, it should be POST & DELETE.
To be clear here- we are talking about the HTTP request being sent to the application here on the /actors endpoint for actor invocation.
|
|
||
| Users today are required to use "normal" Scheduler jobs to ensure message durable execution, and implement their own broadcast or message routing functionality. | ||
| Typically creating a job which has an infinite schedule that triggers at a fixed small interval. | ||
| Moving this functionality as a first concept to Dapr improves performance and improves the developer experience in creating complex distributed systems. |
There was a problem hiding this comment.
How does this improve performance compared to "normal" Scheduler jobs to ensure message durable execution"?
There was a problem hiding this comment.
My take is that today we have "normal" Scheduler jobs for jobs and reminders. This gives us a mechanism to extend the concept to durable scheduled activities (as necessary for actor pub/sub and likely other use cases). I don't know it necessarily improves performance so much as abstract away the downstream purpose from the scheduler itself.
There was a problem hiding this comment.
It improves performance because today, one would have to rely on regular triggering of a job to signal a "keep alive" for the object to exist, without any easy way to signal a message has been deleted. Each trigger needs a round trip through the scheduler queue and call to each client over the network.
This broadcast job type achieves CRUD message behaviour with the least number of Scheduler queue events and network calls .
|
|
||
| https://github.com/dapr/proposals/pull/78 | ||
|
|
||
| ## Background |
There was a problem hiding this comment.
nit: we talk about "durable" a lot, we could define that in the background
There was a problem hiding this comment.
Just about everything is stateless with regards to Dapr. Components are registered with the runtime, but if the runtime shuts down and a component removed, it doesn't have any sort of memory that it ever was. Once a message is brokered through the runtime for PubSub, there's no history to recall of it. Arguably, this is part of the case for the event streaming proposal is that after a workflow lifecycle event occurs, that's it, it's gone - the runtime isn't persisting it anywhere. When we look at streaming pubsub subscriptions, the application registers the gRPC channel with daprd for a given component and topic, but if the application crashes, the channel is closed and that's the end of it.
The difference with Actor PubSub is that when an actor subscribes to a PubSub topic, this time, we need information about that subscription to persist somewhere because if the actor is unloaded, that subscription still needs to be saved somewhere. The distributed scheduler is an excellent place for that because it can hold a little bit of state with each scheduled job, it's already architected to be quite scalable and it's very performant. So, the durability here is in regards to where we can keep this little bit of subscription data where it'll persist between actor invocations and even sidecar restarts (unlike nearly anything else in Dapr, except Jobs and Reminders).
|
|
||
| #### Keep Alive | ||
|
|
||
| The gRPC keep alive options for both the Scheduler server and daprd clients need to be updated so that they are more aggressive. |
There was a problem hiding this comment.
why does this need to change?
There was a problem hiding this comment.
To ensure we have better settings to ensure the health of each stream to Scheduler. This was written more as a TODO for myself to improve the integrity of the client connection pool state. Realistically, this shouldn't have any behavioral changes to the API.
Signed-off-by: joshvanl <[email protected]>
Signed-off-by: joshvanl <[email protected]>
|
+1 binding |
Related: #78