Self-hosting Inngest

[djfarrelly]

This can act as a high level plan and discussion on how we plan to self-host Inngest.

We hope to get input or ideas from those who want to self-host and may have particular underlying systems or infra that they want to run this on.

---

Overview

Updated July 6, 2022

This section will be edited as the discussion evolves and things are improved.

State & infrastructure

Each self-hosted set up is required to have backing infrastructure or services, e.g. databases and queues, to run the Inngest software on top of.

• Event Stream, a message system which acts as an event bus for incoming events
• Job Queue, a queue which schedules and invokes particular steps within a function
• State store, which records state for function runs.
• System store, which manages source keys, function metadata, and action metadata
• An execution runtime, which is where your functions will run, e.g. a container scheduler

APIs & Services

The following Inngest services need to be run in a production set up. Ideally, each services should be horizontally scaled.

• Events API, which accepts events and publishes them to the messaging system
• Runner, which receives events from the messaging system and enqueues functions
• Executor, which executes individual steps within a function.
• Inngest API, which is the external interface for managing the system data, e.g. deploying functions with the CLI

Requirements

The state store, queueing systems, and messaging systems must be highly available. We have multiple drivers for each, and recommend using widely available systems such as hosted, HA Redis, SQS/SNS, or Google Pub/Sub.

---

If you want to chat with the team real time, join our Discord: https://www.inngest.com/discord

[tonyhb]

Some notes:

• It would be great to have an "all in one" state store which manages function state, config. Postgres compat would be ideal here; you could run on psql, cockroachDB, rds, etc. in the cloud.
• We should have a subcommand for running the server: (inngest serve $service) to run each service. We could then provide a single docker which can be used to run inngest in a self-hosted manner.

[djfarrelly]

We're still working out naming and some details on how one might be able to combine parts to use an "all-in-one" store like @tonyhb mentioned, but this is how we're thinking about the architecture right now:

• 4 services to run: Events API (formerly Source API), Runner, Executor, and the Inngest API (open to better names 😁 )
• 4 backing services to run: Event Stream, Job Queue, State DB, System DB
• 1 execution environment to provide, e.g. Nomad, Kubernetes, ECS, Lambda, etc.

Share your feedback if you have any - will update the top level description w/ the updated plans.

[josegonzalez]

Lets say we are running this thing on a generic AWS Kubernetes cluster with AWS tech (I run Nomad, but I figure a k8s explanation would help better for folks). Can ya'll fill in what each component in AWS might be today, as well as what bits can scale in/out and what can't?

For example:

• The Events API/Inngest API/Runner/Executor would all be generic deployments that can scale out as needed. What would be appropriate methods to track scaling for each component? What does resource utilization look like?
• The event stream could be AWS Managed Kafka. Are there other services that might fit here?
• The job queue could be SQS. Same question as for the event stream.
• The state and system DBs could be some RDS instance. How are these used/what should they be provisioned at?
• Are there other bits missing from the stack?

[djfarrelly]

Hey @josegonzalez - you're spot on with your comments and a few cloud "stacks" will definitely be in our docs as we proceed. I'd like an AWS and GCP to start w/ open source services as well that could run anywhere. Here's a reference for what could work for backing components:

Resources

• Event Stream - AWS managed Kafka, SQS, or SNS+SQS, Kinesis
  • Inngest uses Go Cloud's pubsub package so really anything with an existing driver, but really anything could be used if it implements the pubsub package patterns.
  • The simplest set up would be SQS or Kafka
  • An advanced set up might use a SNS+SQS set up for multi-region or multi-account. Think a single global Events API with data being processed in different regions' SQS queues.
• Job Queue - SQS fits best, but you could use managed Redis w/ Redis Streams
• DBs - The State and System DBs could be the same in a simple setup, but for performance when scaling, it would be ideal to split them into databases designed for their purposes:
  • System DB - This stores the configuration of how the system should receive events and execute jobs. The basic information is the function versions, function event triggers, function step versions, Events API keys. Most of the data is immutable as there is versioning, so it is read more than it is written to.
    • AWS RDS w/ pretty much any db engine would work well.
    • DynamoDB would work here as well, although, indexes would have to be set up correctly and it could be expensive depending on your experience provisioning read/write capacity.
    • Theoretically DocumentDB could work as well, but our initial work will be around relational DBs
  • State DB - This stores the current state of jobs being executed. The Runner and Executor write the metadata for current jobs including the function being executed, the event itself and the responses/errors from individual function steps. This state is used to pass data from step to step as well as retry failed steps.
    • AWS ElasticCache (Redis) - This would provide the performance that would be ideal in a production setup.

Services

• Our goal is for all 4 Inngest services to produce metrics that can be used for understanding when to scale out (CPU, throughput, etc.). CPU and throughput should be fine indicators for scaling horizontally.
  • Runner - This may partially depend on your Event Stream's configuration. For example, if you're using Kafka, it would be ideal for your Kafka topic to have enough partitions so you can scale our your Runners (Kafka consumers as part of a consumer group in this case) horizontally.
  • Executor - In the future we may also expose metrics for the "time to execution" which could show how long a function takes to get triggered from the time when the event is received. Your system may have different design goals which may mean you need to scale up your executors (and maybe Runners).

Misc

Other areas not included in the stack or diagram are your own choices in:

• Network - network policy, reverse proxies, load balancers, etc.
• Authentication - Authentication is current out of scope for Inngest API's self-hosted MVP, but we hope to add it in the future. We plan to learn more about how people might want to authenticate
• Image registry - We're leaving choice of and authentication for your container image registry to the user, the Inngest CLI will build and push images, but your user (or machine for CI/CD) will have to be authenticated with your registry. Inngest Cloud run's its own registry proxy and we understand teams may have different desires for how they do this.

---

I know that's a bit of a brain dump here and @tonyhb shared some great thoughts in Discord, but I hope this helps aide in planning and discussion for you and others!

[razzeee]

So can this be legally done? I guess using https://hub.docker.com/r/inngest/inngest (which doesn't seem to have any docs) would be a starting point

[darwin67]

Legally you can self-host, as long as you're not doing whole sale. :)

Unfortunately, it's not the highest priority for us at the moment, and bits and pieces are missing for the OSS version to work as a standalone workload that you can use in prod. e.g. authn, authz etc

We do have self-hosting in mind as a roadmap item so efforts around this will happen, just unclear when yet.

[gavry]

FYI your guys' product is awesome!

As far as self-hosting goes, what are the main things missing?

If it's just auth, would you say it might be usable in prod by hosting within a private network? Alternatively, by self imposing auth before running the server handler? Thanks!