Single-app Heroku deployment

[levic]

(This is a half-Heroku, half-Hasura question:)

Is there a recommended way of doing atomic app deployments with Hasura + an app server (eg django or rails) on Heroku?

In a Heroku Procfile only the 'web' process type can receive HTTP requests. This is a problem if I have an app server and Hasura: I have to create two Heroku apps (one for the app server and one for Hasura) -- I now have to deal with potential race conditions as the apps may not deploy at the same time.

It's actually even worse if I use webhooks: I have to configure 3 apps if I want to avoid thread starvation. Consider a request that goes from app server (frontend request) -> hasura -> app server (webhook callback): what if the frontend app server threads are all blocked waiting on hasura and hasura tries to call a webhook?

If I could combine Hasura and the app into a single Heroku buildpack then this problem wouldn't exist: Each dyno would have its own copy of the frontend app, Hasura, and a second copy of the app purely to deal with webhook callbacks (memory usage might be higher but CPU would be identical since each frontend thread will be blocked waiting on a webhook callback thread anyway)

My understanding of the current build process is:

• The Heroku buildpack is just a wrapper to the hasura docker image
• The hasura docker image Makefile uses a Dockerfile that is based on a single static binary built using fpco/stack-build and copied into a scratch base image

Options to handle this

• Create another docker image based on the hasura one that the app server is installed into, but this would involve somehow merging an app image with the rootfs that hasura uses (or more likely, getting graphql-engine working in a subdirectory and merging that with say a debian image). This could work but packaging would be slow compared to a normal heroku build.
• A much more convenient solution would be to package up graphql-engine as a static binary (see statically compile server binary #724) that could then just be dropped into any heroku buildpack without needing docker at all.

Are there any simpler options that I'm overlooking? Is there a better way of creating single-app Heroku deployments?

[shahidhk]

@levic You could try building a single docker container which runs all applications, Hasura, your app server, the frontend and a gateway to manage URLs. But this is an anti-pattern in the Docker world. Each container is supposed to run one process.

Statically building the binary is not possible at the moment as the ecosystem is still evolving.

Also, I am not sure if I understand the concerns properly. Can you elaborate a little bit on

what if the frontend app server threads are all blocked waiting on hasura and hasura tries to call a webhook
?

[levic]

Also, I am not sure if I understand the concerns properly. Can you elaborate a little bit on

what if the frontend app server threads are all blocked waiting on hasura and hasura tries to call a webhook
?

Consider for simplicity a single-process single-threaded synchronous application server (python/ruby/node/php). A typical setup might look like:

• nginx handles the incoming request from the client
• nginx passes this through via a unix socket to the application server
• the application server makes a http request to hasura using a synchronous graphql library; while waiting for a response from hasura the application server process/thread is blocked
• hasura triggers a webhook request back to port 80
• nginx handles the incoming request and passes it to the application server
• the application server is still blocked waiting for a response from hasura => we now have a deadlock (until something times out and the request fails)

You can increase the number of processes/threads or make it asynchronous, it doesn't make a difference -- as soon as you impose any sort of concurrent request limit then you run the risk of a deadlock.

The only way to guarantee success is to ensure that webhooks are not part of the same concurrent request limit as regular browser requests (eg run a second application server instance on a different port dedicated to handling web hooks only, or use a single application server but configure nginx with separate limits for browser vs webhook requests)

[zenflow]

Also, I am not sure if I understand the concerns properly. Can you elaborate a little bit on

@shahidhk To summarize and generalize @levic's concern here: I believe he is talking about the nastiness we would have if we don't create a third server, separate from the frontend app, to handle hooks for Hasura, and instead we have a frontend which depends on (makes calls to) Hasura which depends on the frontend.

You may be wondering why on earth someone would even consider such an inherently complicated architecture. This goes to show you how strongly some developers want to build their app out as a monolith.

I believe the modern trend of microservice architecture has overshadowed the fact that monolithic architecture still has its advantages (as well disadvantages) over microservices, and depending on your project, monolith may be a better choice.

• advantages of monolithic over microservices:
  • simplified deployments & devops
  • works smoothly with any PaaS provider
  • never a need to update production services in a certain order
  • allows us to effectively use PaaS features like Heroku's Review Apps
  • with some PaaS providers (e.g. Heroku, render) saves the cost of hosting additional "apps" or "services"
  • easier to start the entire system (locally or in CI) for integration & end-to-end testing (manual or automated)
• advantages of microservices over monolithic: This is already well documented. The advantages seems to apply more to "enterprise grade" apps (e.g. apps developed by large and or distributed teams, apps that have high performance requirements) and apps that have many logical components.

---

But this is an anti-pattern in the Docker world. Each container is supposed to run one process.

That does not seem to be entirely true. See this official Docker documentation on running multiple services in a container:

It is generally recommended that you separate areas of concern by using one service per container. [...] It’s ok to have multiple processes, but to get the most benefit out of Docker, avoid one container being responsible for multiple aspects of your overall application.

Why are multi-service containers not generally recommended? I don't have an official answer on this, but judging from the documentation page & knowing how docker works with ENTRYPOINT/CMD, it seems it's just because Docker wasn't really designed with this in mind, and you need to bring in another solution (pattern or tool) and do some work to make it work well. This can be a worth the cost in some cases (e.g. when we really want to build a monolith).

---

That is why I think Hasura should consider & support monolithic architectures. Thanks for reading and thanks for the incredible OSS project! 😀

[zenflow]

Are there any simpler options that I'm overlooking? Is there a better way of creating single-app Heroku deployments?

@levic The approach that @shahidhk suggested #1289 (comment) worked for my Hasura + nodejs monolith.

I am currently working on a npm package (nodejs) to help with both (1) running multiple services in a single container, and (2) creating the http gateway service.

When that's a little more ready, I will share an example project demonstrating the total setup/architecture.

[ro-savage]

@shahidhk - it would be great to have Hasura support monolithes. Currently Hasura doesn't work play nicely with Heroku as per @zenflow comments.

We would love to use hasura with our Heroku setup but it's really a burden since we have to deploy hasura as a seperate app which isn't practically for us.

In Heroku's specific instance, a hasura buildpack would solve the issue. However, a solution that may work for other PaaS would be providing the binary instead of only providing a docker image.

[zenflow]

Create another docker image based on the hasura one that the app server is installed into, but this would involve somehow merging an app image with the rootfs that hasura uses

@levic As of version 2, the Hasura docker image has the apt package manager (i.e. apt-get) so it's easy to base your image on the Hasura image and install whatever you want (e.g. python, node, whatever). Example

Once you have Hasura GraphQL Engine & your app installed in the same image, you can refer to this documentation for options on how to run multiple services/processes in a single container: https://docs.docker.com/config/containers/multi-service_container/