docs: Update references, guides (#3415)

* docs: update BigQuery getting started

* docs: brief refresh of operations guides

* docs: reorder get data sections

* docs: update networks in project files

* docs: update references

* New technical architecture diagram

* fix: link in getting started

Author: ryscheng

apps/docs/docs/get-started/bigquery.mdx

@@ -6,9 +6,9 @@ sidebar_position: 3
 import Link from "@docusaurus/Link";
 import Button from "../../src/components/plasmic/Button";
 
-
-OSO's data lake is currently located in BigQuery on Google Cloud (GCP).
-Every data model is made publicly available by a BigQuery dataset.
+OSO uses BigQuery as the means of sharing large datasets with the community.
+Unless you plan to setup your own data pipeline, we recommend that
+you either start with [pyoso](./python.md) or the [API](./api.mdx) to access OSO data.
 
 This page will guide you through the process of subscribing to the OSO production dataset and making your first query.
 
@@ -77,7 +77,7 @@ Create a linked dataset in your own GCP project.
 Open a new tab by clicking on the `+` icon
 on the top right of the console to `Create SQL Query`.
 
-From here you will be able to write any SQL you'd like any OSO dataset.
+From here you will be able to write any SQL you'd like to any OSO dataset.
 For example, you can query the `oso_production` dataset for
 all available collections like this:
 

apps/docs/docs/guides/ops/gcloud.md

@@ -4,3 +4,11 @@ sidebar_position: 2
 ---
 
 TODO: add guides on how to manage the Google Cloud services
+
+## Kubernetes
+
+## Dagster
+
+## Iceberg / Nessie
+
+## Trino

apps/docs/docs/guides/ops/hasura.md

@@ -3,22 +3,12 @@ title: Hasura
 sidebar_position: 3
 ---
 
-## Setup
-
-When setting up Hasura, the main things that need to be configured are the environment variables.
-For a complete list of environment variables, see the
-[Hasura reference](https://hasura.io/docs/latest/deployment/graphql-engine-flags/reference/).
-
-- Env vars:
-  - `HASURA_GRAPHQL_ADMIN_SECRET`: choose a random password
-  - `HASURA_GRAPHQL_CORS_DOMAIN`: "\*"
-  - `HASURA_GRAPHQL_UNAUTHORIZED_ROLE`: "anonymous"
-  - `HASURA_GRAPHQL_DATABASE_URL`: enter the connection string for the Postgres database
-  - `HASURA_GRAPHQL_AUTH_HOOK`: "https://www.opensource.observer/api/auth"
-- Domains: follow the prompt to enable a custom domain at `api.opensource.observer`.
+We currently use Hasura v3 (DDN) for serving the GraphQL API.
+Hasura will auto-generate the GraphQL schema by introspecting
+the consumer Trino cluster.
 
 ## Metadata
 
 Any configurations outside of environment variables should be set via metadata in the OSO repository.
 For more info on how to do this, check out the
-[README](https://github.com/opensource-observer/oso/tree/main/hasura) under `./apps/hasura`.
+[README](https://github.com/opensource-observer/oso/tree/main/apps/hasura-trino) under `./apps/hasura-trino`.

apps/docs/docs/guides/ops/index.md

@@ -10,6 +10,6 @@ The OSO architecture runs on the following platforms:
 - Vercel: used to build and host the [frontend](https://www.opensource.observer)
 - Cloudflare Pages: used to host the [documentation](https://docs.opensource.observer)
 - [GitHub Actions](https://github.com/opensource-observer/oso/actions): used for CI/CD
-- [Google Cloud](./gcloud)
-- [Hasura](./hasura) - GraphQL API service
-- [Supabase](./supabase) - user authentication and user database
+- [Google Cloud](./gcloud): self-managed k8s for Trino, Iceberg and Dagster
+- [Hasura](./hasura): GraphQL API service
+- [Supabase](./supabase): user authentication and user database

apps/docs/docs/guides/ops/service.md

@@ -12,20 +12,23 @@ that the community aims to achieve together.
 
 ## Service Status
 
+You can find our status page here:
+
+[https://www.opensource.observer/status](https://www.opensource.observer/status)
+
 ## Service Definition
 
 OSO is constantly experimenting with new technology and services
 to improve our infrastructure.
 We only aim to provide service availability for the following mature services:
 
-- [Dagster](https://data.opensource.observer)
-- [dbt pipeline](https://dbt.opensource.observer)
-- [API](https://api.opensource.observer)
+- [Dagster](https://dagster.opensource.observer)
+- [sqlmesh pipeline](https://dagster.opensource.observer/assets/sqlmesh?view=folder)
+- [API](https://www.opensource.observer/graphql)
 - [Frontend](https://www.opensource.observer)
 - [Documentation](https://docs.opensource.observer)
 
-_Note: the sqlmesh pipeline is currently in development and not yet reliable for production use_
-
 ## Outage Escalation
 
-There are 2 surfaces
+The best way to get in touch with the team is via
+[Discord](https://www.opensource.observer/discord).

apps/docs/docs/guides/ops/supabase.md

@@ -3,29 +3,31 @@ title: Supabase
 sidebar_position: 4
 ---
 
+Supabase is the main OLTP database that we use
+for user operations on the website.
+
 ## Setup
 
 - Database >> Extensions
   - Enable the "plpgsql", "plv8" and "pgjwt" extensions
 - Authentication >> Providers
-  - Turn on the Google provider
+  - Turn on the Google and GitHub provider
 - Authentication >> URL Configuration
   - Add allowed redirect URIs
     - `https://www.opensource.observer/**`
     - `http://localhost:3000/**`
+- Authentication >> Hooks
+  - Enable the "hasura_token_hook" hook under the "public" schema
 - Settings >> Authentication
   - Turn on "Allow new users to sign up"
 
 Run the Supabase migrations to create the tables and functions required for the OSO project.
 
 ```bash
-cd frontend/
+cd apps/frontend/
 pnpm supabase db push
 ```
 
-- Authentication >> Hooks
-  - Enable the "hasura_token_hook" hook under the "public" schema
-
 ## Migrations
 
 For updating the database schema or functions, make sure to put it in a migration.

apps/docs/docs/integrate/index.md

@@ -5,9 +5,9 @@ sidebar_position: 0
 
 Open Source Observer is an open data pipeline for measuring the impact of open source efforts. Explore the following resources:
 
-- 📊 [Subscribe to Public Datasets](./datasets/index.mdx) - Get access to any of the public datasets that OSO maintains or builds on top of
 - 🔍 [Explore the Data Lake](./query-data.mdx) - Query the OSO data lake using `pyoso`
 - ⚡ [Use the GraphQL API](./api.md) - Integrate OSO registries and metrics into a live production application
 - 📂 [Import or Clone OSS-Directory](./oss-directory.md) - Leverage oss-directory data separate from OSO
+- 📊 [Subscribe to Public Datasets](./datasets/index.mdx) - Get access to any of the public datasets that OSO maintains or builds on top of
 
 See the [Tutorials](../tutorials/index.md) for more examples and the [Guides](../guides/index.mdx) for more detailed guides for integrating with specific tools.

apps/docs/docs/projects/index.md

@@ -138,22 +138,22 @@ For the latest fields, see the [project schema](https://github.com/opensource-ob
 The simplest way to add all contracts and factories associated with your project is to just add the deployer address in the project file. We will then automatically index all contracts and factories associated with the deployer address. If the deployer is on multiple EVM networks, you can use the `any_evm` tag instead of listing each network individually.
 :::
 
-OSO currently supports the following blockchain networks, which can be enumerated in the `networks` field of a blockchain address:
+If you prefer to enumerate addresses, you should specify the `networks` and `tags`
+for the address.
+Some common `networks` include:
 
 - `any_evm`: Any Ethereum Virtual Machine (EVM) network. This is the recommended tag for EOAs that deploy contracts on multiple EVM networks.
 - `mainnet`: The Ethereum mainnet.
 - `arbitrum_one`: The Arbitrum L2 network.
 - `base`: The Base L2 network.
-- `frax`: The Frax L2 network.
-- `metal`: The Metal L2 network.
-- `mode`: The Mode L2 network.
 - `optimism`: The Optimism L2 network.
-- `zora`: The Zora L2 network.
+- `zksync_eta`: zkSync Era L2 network.
 
 Note: We do not support testnets for any of these networks and do not intend to.
 
 The following tags can be used to describe blockchain addresses:
 
+- `bridge`: A bridge address
 - `deployer`: A deployer address.
 - `eoa`: An externally owned account (EOA) address.
 - `safe`: A multisig safe contract address.
@@ -164,7 +164,10 @@ In previous versions of the schema, we enumerated contracts and factories with t
 - `contract`: A smart contract address.
 - `factory`: A factory contract address.
 
-Read below for more detailed steps on how to add or update project data or consult the [schema](../guides/oss-directory/) for more information.
+For a full list of supported networks and tags, check out the
+[schema on oss-directory](https://github.com/opensource-observer/oss-directory/blob/main/src/resources/schema/blockchain-address.json).
+
+Read below for more detailed steps on how to add or update project data or consult the [guide](../guides/oss-directory/) for more information.
 
 ## Add or Update Project Data (Detailed Steps)
 

apps/docs/docs/references/architecture.md

@@ -18,7 +18,7 @@ OSO maintains an [ETL](https://en.wikipedia.org/wiki/Extract%2C_load%2C_transfor
 - **Load**: the results into various OSO data products (e.g., our API, website, widgets)
 
 The following diagram illustrates Open Source Observer's technical architecture.
-[![OSO Architecture](https://mermaid.ink/img/pako:eNqVVU1z2jAQ_SsancwktNMcOXSGhHbaTgikNOkB9yDLC1ZjS44-gDST_96VLXDAhja-gFfvzdt9u149U65SoAPa7_djaYXNYUBiOilBkplymgOZJAb0CjQZap4JC9w6DTGNZUWJ5SJXa54xbcmPy1gSfIxLlpqVGSldkgs-j-mQW6UNiW6U7MMmY85YsYJeTH_VDP84lDGIvfO_eycprPzBiFlGPsmlkHCIYJLlT8buUDMuAKvByA4GMq3_VDqk38cy7wWsyXty60A_xRRjH4kyag1JjfS6NfAAwUpxiLhS0mqROAvEOxqwQqawAV2Dt0m2CLPb64DnqigxkjDLsy7SdyfJI-YiwATGmmnIFNZUww8agLnOu7tZ-dIY2LRMlJCjwx-QN2ZCksrPaYjume4fhTMBxmpmhZKe800l-0ESjdjSWNC9Fjm4s20aV3kOYVDS3Lbxu1rnUaDsIiS6FMuqS0jrHfBeuzoPzSSlVhyMqdUSezS7VrdSFO42v0uvpo8ZGiBYLv7AaWpTUMWb1kkenY7dYDev_yPuS0iY1z4xARdo1WeNdaPAW6bg4o1TsE3GN3VyPZxWWj5AoivcHg-VHR1dNY95Aaa7oeHsuFiXvYHUqfJvJ4-0o94W7S1ysgWLYPvrDvjPVnAwbfOrjYXQn5AQVpboWbD8HvCLz89uYGPf_Ta9Libm5tfz9CsJSz76wozT7GxY4teoekfUTm_FvXXbbKXmHx7Sc1qALphI8fJ59uGY2gwKNNZfQCnTD_6OeUEcc1bNniSnA6sdnFOt3DKjgwXLDb65Eq2EkWBoXbGFQCpwj4zrq6264V7-AjpFJyI?type=png)](https://mermaid.live/edit#pako:eNqVVU1z2jAQ_SsancwktNMcOXSGhHbaTgikNOkB9yDLC1ZjS44-gDST_96VLXDAhja-gFfvzdt9u149U65SoAPa7_djaYXNYUBiOilBkplymgOZJAb0CjQZap4JC9w6DTGNZUWJ5SJXa54xbcmPy1gSfIxLlpqVGSldkgs-j-mQW6UNiW6U7MMmY85YsYJeTH_VDP84lDGIvfO_eycprPzBiFlGPsmlkHCIYJLlT8buUDMuAKvByA4GMq3_VDqk38cy7wWsyXty60A_xRRjH4kyag1JjfS6NfAAwUpxiLhS0mqROAvEOxqwQqawAV2Dt0m2CLPb64DnqigxkjDLsy7SdyfJI-YiwATGmmnIFNZUww8agLnOu7tZ-dIY2LRMlJCjwx-QN2ZCksrPaYjume4fhTMBxmpmhZKe800l-0ESjdjSWNC9Fjm4s20aV3kOYVDS3Lbxu1rnUaDsIiS6FMuqS0jrHfBeuzoPzSSlVhyMqdUSezS7VrdSFO42v0uvpo8ZGiBYLv7AaWpTUMWb1kkenY7dYDev_yPuS0iY1z4xARdo1WeNdaPAW6bg4o1TsE3GN3VyPZxWWj5AoivcHg-VHR1dNY95Aaa7oeHsuFiXvYHUqfJvJ4-0o94W7S1ysgWLYPvrDvjPVnAwbfOrjYXQn5AQVpboWbD8HvCLz89uYGPf_Ta9Libm5tfz9CsJSz76wozT7GxY4teoekfUTm_FvXXbbKXmHx7Sc1qALphI8fJ59uGY2gwKNNZfQCnTD_6OeUEcc1bNniSnA6sdnFOt3DKjgwXLDb65Eq2EkWBoXbGFQCpwj4zrq6264V7-AjpFJyI)
+[![](https://mermaid.ink/img/pako:eNqFVU1P3DAQ_SuWT1nBtlKPe6gERVWpykcLpVJJD44zJC6OHcY2C0X8947jLNlsdiGXJPa88Zv3MpMnLm0JfMHn83luvPIaFiznZy0YdmEDSmBnhQO8B2QHKGvlQfqAkPPcdJDc3Gi7lLVAzy4Pc8PocqGoULQ1a0OhlbzO-YH0Fh3LTq2Zw0MtgvPqHmY5_5MQ8Qp0jKPYn_E-2inhPm4cCS-YkwqIp_OOvWdgKmVgPRxMmR66bGw-p2KuFCwp-HsAfMw5rX1k1tklFCkyZk-Bn6zxqIrggUVV-lhlSngAfCW4oWDtVqlJJXAehVfWbIJ-BMPuiIcCNzARrUpxG-rR1vV2K7pyB30GvVULmhQh2IlQhnWKnfeLI0njNWJKkK-2GK-x7EhUzgPuuTvdgKtnkxy9OCt3pNUaeq9L7afxhapi_Y_XWY9YCoTaklssO1RV5xGhZhswaZuWlC6ElzWddSjkLTm9WmZSh0iTZZeojJ1tyDOpNXlB5a5MWE-_tb6J5SVx78Fa3KYKdlNO8BNBHJXQ6h-8Dl2JlGDnaCW4t6m-JHsbtgMYTfn2kiU7llAAVlvtMC40gD7KTX5c0FepTMUS69SVIzN2ghPX9dbcoknX1MNrapkpcpR4a3_cIDlI2Yjy5_6RxY5SEty0PboZQaG_oGCibWmS9V1xBdSMeu8UHvy7v262DUkE49g7P2b98My-CBdQ7B201CJ2tuO0aVHr82E04IaBMTzRJt_nJEEjVElD_Sku59zX0NAnFwd7KfA2zu5nihPB24tHI_nCY4B9jjZUNV_cCO3oLbT0jcOREiRd87LaCvPb2uEdSkXdfpL-Id2vZJ9XGA_vcxIlwE82GM8XH57_A664BIU?type=png)](https://mermaid.live/edit#pako:eNqFVU1P3DAQ_SuWT1nBtlKPe6gERVWpykcLpVJJD44zJC6OHcY2C0X8947jLNlsdiGXJPa88Zv3MpMnLm0JfMHn83luvPIaFiznZy0YdmEDSmBnhQO8B2QHKGvlQfqAkPPcdJDc3Gi7lLVAzy4Pc8PocqGoULQ1a0OhlbzO-YH0Fh3LTq2Zw0MtgvPqHmY5_5MQ8Qp0jKPYn_E-2inhPm4cCS-YkwqIp_OOvWdgKmVgPRxMmR66bGw-p2KuFCwp-HsAfMw5rX1k1tklFCkyZk-Bn6zxqIrggUVV-lhlSngAfCW4oWDtVqlJJXAehVfWbIJ-BMPuiIcCNzARrUpxG-rR1vV2K7pyB30GvVULmhQh2IlQhnWKnfeLI0njNWJKkK-2GK-x7EhUzgPuuTvdgKtnkxy9OCt3pNUaeq9L7afxhapi_Y_XWY9YCoTaklssO1RV5xGhZhswaZuWlC6ElzWddSjkLTm9WmZSh0iTZZeojJ1tyDOpNXlB5a5MWE-_tb6J5SVx78Fa3KYKdlNO8BNBHJXQ6h-8Dl2JlGDnaCW4t6m-JHsbtgMYTfn2kiU7llAAVlvtMC40gD7KTX5c0FepTMUS69SVIzN2ghPX9dbcoknX1MNrapkpcpR4a3_cIDlI2Yjy5_6RxY5SEty0PboZQaG_oGCibWmS9V1xBdSMeu8UHvy7v262DUkE49g7P2b98My-CBdQ7B201CJ2tuO0aVHr82E04IaBMTzRJt_nJEEjVElD_Sku59zX0NAnFwd7KfA2zu5nihPB24tHI_nCY4B9jjZUNV_cCO3oLbT0jcOREiRd87LaCvPb2uEdSkXdfpL-Id2vZJ9XGA_vcxIlwE82GM8XH57_A664BIU)
 
 ## Major Components
 
@@ -27,48 +27,41 @@ The architecture has the following major components.
 ### Data Orchestration
 
 Dagster is the central data orchestration system, which manages the entire pipeline,
-from the data ingestion (e.g. via [dlt](https://docs.dagster.io/integrations/embedded-elt/dlt) connectors), the [dbt](https://docs.dagster.io/integrations/dbt) pipeline, the [sqlmesh](https://github.com/opensource-observer/dagster-sqlmesh) pipeline, to copying mart models to data serving infrastructure.
+from the data ingestion (e.g. via [dlt](https://docs.dagster.io/integrations/embedded-elt/dlt) connectors) to the [sqlmesh](https://github.com/opensource-observer/dagster-sqlmesh) pipeline.
 
 You can see our public Dagster dashboard at
 [https://dagster.opensource.observer/](https://dagster.opensource.observer/).
 
-### Data Warehouse
+### Data Lakehouse
 
-Currently all data is stored and processed in
-[Google BigQuery](https://cloud.google.com/bigquery/?hl=en).
-All of the collected data or aggregated views used by OSO is also made publicly available here (if it is not already a public dataset on BigQuery).
-Anyone with can view, query, or build off of any stage in the pipeline.
-In the future we plan to explore a decentralized lakehouse.
+Currently all data is stored in managed Iceberg tables.
 
-To see all datasets that you can subscribe to, check out our
+We also make heavy use of public datasets from
+[Google BigQuery](https://cloud.google.com/bigquery/?hl=en).
+To see all BigQuery datasets that you can subscribe to, check out our
 [Data Overview](../integrate/datasets/index.mdx).
 
-### dbt pipeline
+### sqlmesh pipeline
 
-We use a [dbt](https://www.getdbt.com/) pipeline to clean and normalize the data
-into a universal event table. You can read more about our event model
+We use a [sqlmesh](https://sqlmesh.com/) pipeline to clean and normalize the data
+into a universal event table and metrics. You can read more about our event model
 [here](./event.md).
 
-### OLAP database
+### Trino clusters
 
-We use [Clickhouse](https://clickhouse.com/)
-as a frontend database for serving live queries to the API server
-and frontend website, as well as running a sqlmesh data pipeline.
-
-### sqlmesh pipeline
+We maintain separate Trino clusters the operate over the Iceberg tables:
 
-A [sqlmesh](https://sqlmesh.com/) pipeline
-is used for computing time series metrics from
-the universal event table, which is copied from the BigQuery dbt pipeline.
+- Production pipeline cluster - a read-write cluster to run the sqlmesh pipeline
+- Consumer query cluster - a read-only cluster to serve the API and `pyoso`
 
 ### API service
 
 We use [Hasura](https://hasura.io/) to automatically generate
-a GraphQL API from our Clickhouse database.
+a GraphQL API from our consumer Trino cluster.
 We then use an [Apollo Router](https://www.apollographql.com/docs/router/)
 to service user queries to the public.
-The API can be used by external developers to integrate insights from OSO.
-Rate limits or cost sharing subscriptions may apply to its usage depending
+The API can be used by external developers to integrate data from OSO.
+Rate limits or subscription pricing may apply to its usage depending
 on the systems used. This also powers the OSO website.
 
 ### OSO Website
@@ -81,15 +74,15 @@ hosted by [Vercel](https://vercel.com/).
 
 ## Open Architecture for Open Source Data
 
-The architecture is designed to be fully open to maximum open source collaboration.
+The architecture is designed to be fully open to maximize open source collaboration.
 With contributions and guidance from the community,
 we want Open Source Observer to evolve as we better understand
 what impact looks like in different domains.
 
 All code is open source in our
 [monorepo](https://github.com/opensource-observer/oso).
-All data, including every stage in our pipeline, is publicly available on
-[BigQuery](https://console.cloud.google.com/bigquery/analytics-hub/exchanges/projects/87806073973/locations/us/dataExchanges/open_source_observer_190181416ae).
+All data, including every stage in our pipeline, is publicly available via
+[pyoso](../get-started/python.md).
 All data orchestration is visible in our public
 [Dagster dashboard](https://dagster.opensource.observer/).
 

apps/docs/docs/references/ecosystems/arbitrum.md

@@ -8,11 +8,12 @@ Arbitrum One is an EVM-equivalent Optimistic Rollup chain. It inherits Ethereum-
 
 ## Coverage on Open Source Observer
 
-Open Source Observer is in the process of adding the first collection of open source projects that are building on Arbitrum One. Project data will include Arbitrum addresses associated with smart contracts, deployers, and grant wallets.
+Open Source Observer is in the process of adding open source projects that are building on Arbitrum One. Project data will include Arbitrum addresses associated with smart contracts, deployers, and grant wallets.
 
-The main sources of project data come from [STIP](https://forum.arbitrum.foundation/t/arbitrums-short-term-incentive-program-arbitrum-improvement-proposal/16131), grant programs, and the Dune contracts library.
-
-We will be releasing our first version of Arbitrum One coverage in the coming weeks.
+The main sources of project data come from
+[STIP](https://forum.arbitrum.foundation/t/arbitrums-short-term-incentive-program-arbitrum-improvement-proposal/16131),
+[Stylus sprint](https://blog.arbitrum.io/stylus-sprint/),
+other grant programs, and the Dune contracts library.
 
 ## Ecosystem Resources