PACKAGES: PAAS Engine Design

[joe-getcouragenow]

We can actually use protobufs for the storage layer.

BigQuery does this BTW..

Mongo DB
dataform-co/dataform#542

• Allows Protobufs to be stored in MongoDB as BSON (https://www.mongodb.com/blog/post/bson)
• SQL template that respects the protobufs is needed
• MongoDB can do CDC

Then what to do with the events ?

A highly flexible stream processors that is configurable on the fly makes the system highly extensible.

Streams ETL and CDC, etc
https://www.benthos.dev/
https://github.com/Jeffail/benthos
Web IDE: https://github.com/benthosdev/benthos-lab

• is WASM and so DRY
  https://github.com/Jeffail/benthos/blob/master/cmd/tools/benthos_config_gen/main.go
• config generator. Can also work off Protobufs to gen all docs and other things

Benthos transforms can be designed with the Web GUI, and tested, and then compiled (as WASM at runtime into the Vector runtime.

Vector is a Data change processor that is high perf ( RUST)
vectordotdev/vector#2341

• Pluggable now with WASM just like Envoy is.
• "Introduce plugin support to Vector, providing a way for users to add custom functionality to Vector. This RFC introduces
  WASM plugins specifically, but we envision the possibility of other implementations that may reuse ideas from this RFC.
  This RFC introduces Transforms and will be later amended with Source and Sink functionality."
• TinyGO WASM compiler would be preferable !!!

---

https://github.com/facebookincubator/ent

• Looks much more powerful in the long term
• very high dev productivity
• high support
• Protobufs can be used to generate the schema using golang templates
  • The relations are then added by hand.
• Hooks API, then makes the CDC redundant i think.
  • Can also gen those from the protobufs and send to nats
  • Can also do proper GraphQL subscriptions: https://github.com/pkapatsos-blinq/Magma/blob/master/symphony/graph/graphql/resolver/common_test.go
• Code gen of GraphQL struct tags
  • https://github.com/facebookincubator/magma/blob/master/symphony/pkg/ent-contrib/entgqlgen/internal/todo/ent/schema/todo.go#L30
• Integration to openconsensus
  • https://github.com/facebookincubator/magma/blob/master/symphony/graph/viewer/viewer.go#L20
• authz is designed in
  • https://github.com/facebookincubator/magma/tree/master/symphony/graph/authz
• PUB SUB Support
  • magma/magma@8a49ea3

---

NATS
https://github.com/google/go-cloud/blob/master/pubsub/natspubsub/go.mod

• Integration to openconsensus
• Agnostic interface so can swap out if needed or run on cloud providers own PUB SUB

[Hoverbear]

Hey! I see you're a non-profit doing climate change work! I am doing some of the WASM/protobuf work over at Vector and would be happy to chat with you and help provide any pointers. :)

[joe-getcouragenow]

@Hoverbear Hey and its really nice to hear support from you :)

I better check if your offering advice or thinking about helping us actually integrate Vector into our Platform ? I don't want to take the whole arm when a hand if offered !

I read the docs and code of Vector. Best docs i have ever seen btw. Hats off to you and the team.

Uses i see for Vector

• Logging agent that does filtering for privacy aspects
• Metrics
• CDC with NATS. Saw that Rust NATS is almost read.
  • We currently use TIDB CDC ( https://github.com/pingcap/ticdc ) , but i think being able to configure the "router" of Vector at runtime is really useful. Reminds me of How we use Envoy for GRPC-Web and Authn and Authz
  • Using WASM ( like envoy now has btw ) to make it easy to extend for CDC
• Kubernetes logging
  • We run on GCP, but will move to microk8 ( https://github.com/ubuntu/microk8s ) or lokomotive ( https://github.com/kinvolk/lokomotive ) so we can run on our own Servers soon.

[Hoverbear]

I can help a bit with integration of Vector if it's appropriate for you! :) @binarylogic did a lot of the docs!

It sounds like you've got the rest of things figured out pretty good! :) Are you setting up a grafana/prometheus system for your metrics? Vector can use log_to_metric to automatically add metrics to naughty services through their logs.