examples overhaul; pager option; more link normalization (#1306)

* Snowflake data loader example

* new examples index

* edit examples README

* edit examples README

* examples in sidebar; pager option

* edit examples README

* emoji headings

* brush to zoom

* edit examples README

* checkpoint README

* no link: for example package.json

* pager groups

* document pager

* two markdown-it examples

* markdown-it-container example

* snowflake example edits

* more example cleanup

* more example clean-up

* more example clean-up

* PostgreSQL data loader example

* add “| Observable Framework” to page titles

* consistent title

* IntersectionObserver example

* loader-arrow example

* loader-parquet example

* simplify snowflake query

* deactivate

* more path normalization

* private examples (by default)

* tweak hello-world

* more consistent example READMEs

* Update README

* Update README

* Update README

* Update README

* Update README

* Update README

* custom-stylesheet example

* fix source link

* remove interactive.md

* don’t build examples

* loader-google-analytics example

* loader improvements

* loader-github example

* custom-input-2d example

* add vega-responsive example

* loader-databricks example; other improvements

* Update examples/custom-stylesheet/src/index.md

Co-authored-by: Philippe Rivière <[email protected]>

* Update examples/api/README.md

Co-authored-by: Philippe Rivière <[email protected]>

* Update examples/README.md

Co-authored-by: Philippe Rivière <[email protected]>

* links

* showcase thumbnails

* smaller thumbnail

* target=_blank

* Revert "target=_blank"

This reverts commit 9067d97.

* link to live examples

* import vlresize

* vlresize improvements

* link trailing slash

* adopt octokit

* links and edits

* fix pointerdown

* more improvements

* Update README.md

Typo; unclosed a tag

* -chess

* eia edits

* google analytics edits

* pmms edits (the interaction has changed)

* penguins are classified based on all the numerical fields

* vega-lite static width

* vega-dark example

---------

Co-authored-by: Philippe Rivière <[email protected]>
Co-authored-by: Jeff Pettiross <[email protected]>

Author: 3 people

.github/workflows/deploy.yml

@@ -21,59 +21,16 @@ jobs:
         with:
           node-version: 20
           cache: 'yarn'
-          cache-dependency-path: |
-            yarn.lock
-            'examples/*/yarn.lock'
-      - uses: actions/setup-python@v5
-        with:
-          python-version: '3.12'
-          cache: 'pip'
-          cache-dependency-path: 'examples/*/requirements.txt'
       - run: yarn --frozen-lockfile
       - id: date
         run: echo "date=$(TZ=America/Los_Angeles date +'%Y-%m-%d')" >> $GITHUB_OUTPUT
       - id: cache-data
         uses: actions/cache@v4
         with:
-          path: |
-            docs/.observablehq/cache
-            examples/*/src/.observablehq/cache
-          key: data-${{ hashFiles('docs/data/*', 'examples/*/src/data/*') }}-${{ steps.date.outputs.date }}
+          path: docs/.observablehq/cache
+          key: data-${{ hashFiles('docs/data/*') }}-${{ steps.date.outputs.date }}
       - run: yarn build
       - run: yarn docs:build
-      - name: Build example "api"
-        run: yarn --frozen-lockfile && yarn build
-        working-directory: examples/api
-      - name: Build example "chess"
-        run: yarn --frozen-lockfile && yarn build
-        working-directory: examples/chess
-      - name: Build example "eia"
-        run: yarn --frozen-lockfile && yarn build
-        working-directory: examples/eia
-        env:
-          EIA_KEY: ${{ secrets.EIA_KEY }}
-      - name: Build example "google-analytics"
-        run: yarn --frozen-lockfile && yarn build
-        working-directory: examples/google-analytics
-        env:
-          GA_PROPERTY_ID: ${{ vars.GA_PROPERTY_ID }}
-          GA_CLIENT_EMAIL: ${{ secrets.GA_CLIENT_EMAIL }}
-          GA_PRIVATE_KEY: ${{ secrets.GA_PRIVATE_KEY }}
-      - name: Build example "hello-world"
-        run: yarn --frozen-lockfile && yarn build
-        working-directory: examples/hello-world
-      - name: Build example "mortgage-rates"
-        run: yarn --frozen-lockfile && yarn build
-        working-directory: examples/mortgage-rates
-      - name: Build example "penguin-classification"
-        run: pip install -r requirements.txt && yarn --frozen-lockfile && yarn build
-        working-directory: examples/penguin-classification
-      - name: Build example "plot"
-        run: yarn --frozen-lockfile && yarn build
-        working-directory: examples/plot
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-      - run: mkdir docs/.observablehq/dist/examples && for i in examples/*; do if [ -d $i/dist ]; then mv -v $i/dist docs/.observablehq/dist/examples/$(basename $i); fi; done
       - uses: cloudflare/pages-action@1
         with:
           apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}

docs/config.md

@@ -118,6 +118,8 @@ Both pages and sections have a **name**, which typically corresponds to the page
 
 Sections may be **collapsible**. <a href="https://github.com/observablehq/framework/releases/tag/v1.6.0" class="observablehq-version-badge" data-version="^1.6.0" title="Added in 1.6.0"></a> If the **open** option is set, the **collapsible** option defaults to true; otherwise it defaults to false. If the section is not collapsible, the **open** option is ignored and the section is always open; otherwise, the **open** option defaults to true. When a section is collapsible, clicking on a section header opens or closes that section. A section will always be open if the current page belongs to that section.
 
+Pages and sections may also have a **pager** field <a href="https://github.com/observablehq/framework/pull/1306" class="observablehq-version-badge" data-version="prerelease" title="Added in #1306"></a> which specifies the name of the page group; this determines which pages are linked to via the previous and next pager buttons. If the **pager** field is not specified, it defaults the current section’s **pager** field, or to `main` for top-level pages and sections. (The home page is always in the `main` pager group.) The **pager** field can be also set to `null` to disable the pager on a specific page or section, causing adjacent pages to skip the page.
+
 For example, here **pages** specifies two sections and a total of four pages:
 
 ```js run=false

docs/index.md

@@ -129,7 +129,7 @@ index: false
     </picture>
     <div class="small arrow">Observable Plot downloads</div>
   </a>
-  <a href="https://observablehq.com/framework/examples/mortgage-rates/interactive" target="_blank">
+  <a href="https://observablehq.com/framework/examples/mortgage-rates/" target="_blank">
     <picture>
       <source srcset="./assets/mortgage-rates.webp" media="(prefers-color-scheme: dark)">
       <img src="./assets/mortgage-rates-dark.webp">

examples/.gitignore

@@ -1,2 +1,3 @@
+/*/src/.observablehq/deploy.json
 package-lock.json
 yarn.lock

examples/README.md

@@ -1,14 +1,111 @@
 # Observable Framework examples
 
-*First:* [Get started with Observable Framework](https://observablehq.com/framework/getting-started)
-
-*This repository shows different flavors and features of what you can build with Framework:*
-
-- **API** — Millions of API requests, stored in Arrow format, are analyzed in a report. [[code](./api)] [[live project](https://observablehq.com/framework/examples/api/)]
-- **Chess** — A data loader that retrieves several large zip files from the FIDE, extracts information about the top 10 players, and displays a rank chart. [[code](./chess)] [[live project](https://observablehq.com/framework/examples/chess/)]
-- **EIA** — JavaScript data loaders retrieve hourly data from the U.S. Energy Information Administration API to produce a map and time series charts. [[code](./eia)] [[live project](https://observablehq.com/framework/examples/eia/)]
-- **Google analytics** — Load data from the G.A. API and analyze your website’s traffic. [[code](./google-analytics)] [[live project](https://observablehq.com/framework/examples/google-analytics/)]
-- **Hello, world!** — A minimal test. [[code](./hello-world)] [[live project](https://observablehq.com/framework/examples/hello-world/)]
-- **Plot** — TypeScript data loaders that access metadata from the Observable Plot Github repo. [[code](./plot)] [[live project](https://observablehq.com/framework/examples/plot/)]
-- **Mortgage rates** — A small dashboard reproducing Freddie Mac’s historic mortgage rates dashboard. [[code](./mortgage-rates)] [[live project](https://observablehq.com/framework/examples/mortgage-rates/)][[interactive version](https://observablehq.com/framework/examples/mortgage-rates/interactive)]
-- **Penguin classification** — In a Python data loader, we use logistic regression to predict penguin species based on body size measurements. [[code](./penguin-classification)] [[live project](https://observablehq.com/framework/examples/penguin-classification/)]
+> [!NOTE]
+> To get started with Framework, please read [_Getting started_](https://observablehq.com/framework/getting-started).
+
+## Showcase examples 🖼️
+
+### [`api`](https://observablehq.observablehq.cloud/framework-example-api/) - Analyzing web logs
+
+<a href="https://observablehq.observablehq.cloud/framework-example-api/"><img src="../docs/assets/api.webp" alt="Analyzing web logs" width="312" height="237"></a>
+
+[View source](./api) · This report visualizes millions of requests to Observable’s API servers over a 7-day period in January 2024, revealing both traffic patterns and performance characteristics of Observable’s web service. This example showcases the flexibility of Observable Plot for creating custom, performant visualizations, and hints at the potential of Framework’s data loaders for working with large datasets. This example also demonstrates reading [Apache Parquet files](https://observablehq.com/framework/lib/arrow). (While this public example uses static data, at Observable we use [Snowflake data loaders](https://observablehq.observablehq.cloud/framework-example-loader-snowflake/) internally to create a similar live dashboard.)
+
+### [`eia`](https://observablehq.observablehq.cloud/framework-example-eia/) - U.S. electricity grid
+
+<a href="https://observablehq.observablehq.cloud/framework-example-eia/"><img src="../docs/assets/eia.webp" alt="U.S. electricity grid" width="312" height="237"></a>
+
+[View source](./eia) · This dashboard visualizes electricity generation and demand across the U.S. electricity grid. The included data loaders demonstrate how to retrieve live data from the U.S. Energy Information Administration (EIA) API, while the dashboard demonstrates how to produce interactive maps, bar charts, and time-series charts with Observable Plot. A range input allows the user to rewind time to any point in the previous 24 hours, and a table shows details.
+
+### [`plot`](https://observablehq.observablehq.cloud/framework-example-plot/) - Observable Plot downloads
+
+<a href="https://observablehq.observablehq.cloud/framework-example-plot/"><img src="../docs/assets/plot.webp" alt="Observable Plot downloads" width="312" height="237"></a>
+
+[View source](./plot) · This dashboard visualizes the popularity and development of [Observable Plot](https://observablehq.com/plot/), our open-source visualization library. The included data loaders demonstrate how to retrieve data from GitHub and npm APIs, including star counts, releases, downloads, and open issues. A time-series chart shows daily npm downloads with 7- and 28-day moving averages, and a burndown chart shows the age of open issues over time.
+
+### [`mortgage-rates`](https://observablehq.observablehq.cloud/framework-example-mortgage-rates/) - Primary mortgage market survey
+
+<a href="https://observablehq.observablehq.cloud/framework-example-mortgage-rates/"><img src="../docs/assets/mortgage-rates.webp" alt="Primary mortgage market survey" width="312" height="237"></a>
+
+[View source](./mortgage-rates) · This dashboard visualizes Freddie Mac’s historical mortgage rates data. The included data loader demonstrates how to retrieve CSV data from Freddie Mac and visualize the result as a zoomable chart with Observable Plot. The larger time-series line chart at the bottom allows brushing to select an arbitrary time range, while the smaller visualization above zooms to show the selected range.
+
+## Technique examples 🛠️
+
+### Charts
+
+* [`vega-dark`](https://observablehq.observablehq.cloud/framework-example-vega-dark/) - Responsive dark mode in Vega-Lite
+* [`vega-responsive`](https://observablehq.observablehq.cloud/framework-example-vega-responsive/) - Responsive width in Vega-Lite using ResizeObserver
+
+### Data loaders
+
+* [`loader-arrow`](https://observablehq.observablehq.cloud/framework-example-loader-arrow/) - Generating Apache Arrow IPC files
+* [`loader-databricks`](https://observablehq.observablehq.cloud/framework-example-loader-databricks/) - Loading data from Databricks
+* [`loader-github`](https://observablehq.observablehq.cloud/framework-example-loader-github/) - Loading data from GitHub
+* [`loader-google-analytics`](https://observablehq.observablehq.cloud/framework-example-loader-google-analytics/) - Loading data from Google Analytics
+* [`loader-parquet`](https://observablehq.observablehq.cloud/framework-example-loader-parquet/) - Generating Apache Parquet files
+* [`loader-postgres`](https://observablehq.observablehq.cloud/framework-example-loader-postgres/) - Loading data from PostgreSQL
+* [`loader-snowflake`](https://observablehq.observablehq.cloud/framework-example-loader-snowflake/) - Loading data from Snowflake
+
+### Inputs
+
+* [`custom-input-2d`](https://observablehq.observablehq.cloud/framework-example-custom-input-2d/) - A custom 2D input with bidirectional binding
+
+### Markdown
+
+* [`markdown-it-container`](https://observablehq.observablehq.cloud/framework-example-markdown-it-container/) - The markdown-it-container plugin
+* [`markdown-it-footnote`](https://observablehq.observablehq.cloud/framework-example-markdown-it-footnote/) - The markdown-it-footnote plugin
+* [`markdown-it-wikilinks`](https://observablehq.observablehq.cloud/framework-example-markdown-it-wikilinks/) - The markdown-it-wikilinks plugin
+
+### Other
+
+* [`chess`](https://observablehq.observablehq.cloud/framework-example-chess/) - Loading Zip data from FIDE; creating a bump chart with Observable Plot
+* [`custom-stylesheet`](https://observablehq.observablehq.cloud/framework-example-custom-stylesheet/) - Defining a custom stylesheet (custom theme)
+* [`google-analytics`](https://observablehq.observablehq.cloud/framework-example-google-analytics/) - A Google Analytics dashboard with numbers and charts
+* [`hello-world`](https://observablehq.observablehq.cloud/framework-example-hello-world/) - A minimal Framework project
+* [`intersection-observer`](https://observablehq.observablehq.cloud/framework-example-intersection-observer/) - Scrollytelling with IntersectionObserver
+* [`penguin-classification`](https://observablehq.observablehq.cloud/framework-example-penguin-classification/) - Logistic regression in Python; validating models with Observable Plot
+
+## About these examples
+
+We offer two types of examples: **techniques** and **showcases**.
+
+Technique examples address lower-level needs such as “how to load data from Google Analytics” or “how to make a bump chart”. They’re smaller, piecemeal examples for common development tasks. Technique examples are intended to teach you how to accomplish a specific task and to provide reusable code that can be copy-pasted into your Framework project.
+
+Showcase examples, in contrast, address higher-level user needs such as “how to analyze website traffic” or “how to show the growth of an open-source project”. These larger, complete examples demonstrate how to create useful data apps. Showcase examples are intended primarily to inspire and show Framework’s potential. As applied examples, showcase examples also demonstrate multiple techniques working together; we encourage you to view source.
+
+### How to use these examples
+
+You can browse the source code for any of the examples by navigating to the respective folder on GitHub. You can then copy-paste the code into your own project or download individual files. All of the example code in this repository is covered by the same [open-source license](../LICENSE) as Framework itself.
+
+If you’d like to run (and tinker with) these examples locally, you can [clone the repo](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) like so:
+
+```sh
+git clone [email protected]:observablehq/framework.git
+```
+
+Then `cd` into the desired example directory. From there you would typically run `npm install` or `yarn` to install dependencies. Please refer to each example’s `README.md` file for specific instructions; some examples may require additional configuration, such as setting environment variables to talk to external data sources.
+
+### Can’t find what you need? 🧐
+
+If there’s an example you’d like to see, please let us know by [filing an issue](https://github.com/observablehq/framework/issues).
+
+If you have questions about an existing example, please [open a discussion](https://github.com/observablehq/framework/discussions).
+
+### We welcome contributions! 🤗
+
+If you have an example that you’d like to share with the community, please [open a pull request](https://docs.github.com/en/pull-requests). Please follow the conventions set by the existing examples and let us know if you have any questions.
+
+Here are some technique examples we’d like to see:
+
+* Visualization
+  * Big number with area chart
+  * Daily metric chart with moving average
+  * Punchcard chart (activity by day of week and hour of day)
+  * Bump chart/rank chart
+  * Brushing
+  * Zooming
+* Data loaders
+  * JSZip data loader
+  * npm data loader
+* Markdown
+  * Inline TeX `$…$`

examples/api/.gitignore

@@ -1,6 +1,4 @@
 .DS_Store
-dist/
-src/.observablehq/cache/
+/dist/
 node_modules/
 yarn-error.log
-yarn.lock

examples/api/README.md

@@ -1,14 +1,51 @@
-# API logs
+[Framework examples →](../)
 
-This is an example Observable Framework project. It uses API logs data collected from [observablehq.com](observablehq.com), and visualizes them as heatmaps, histograms and bar charts.
+# Analyzing web logs
 
-View the [live project](https://observablehq.com/framework/examples/api/). 
+View live: <https://observablehq.observablehq.cloud/framework-example-api/>
 
-## Data loaders
+This report visualizes millions of requests to Observable’s API servers over a 7-day period in January 2024, revealing both traffic patterns and performance characteristics of Observable’s web service.
 
-The datasets used are static snapshots. We use an [Apache Arrow](https://arrow.apache.org/) file to handle the large number of API logs.
+This example showcases the flexibility of Observable Plot for creating custom, performant visualizations, and hints at the potential of Framework’s data loaders for working with large datasets. This example also demonstrates reading [Apache Parquet files](https://observablehq.com/framework/lib/arrow). (While this public example uses static data, at Observable we use [Snowflake data loaders](https://observablehq.observablehq.cloud/framework-example-loader-snowflake/) internally to create a similar live dashboard.)
 
-## Charts
+> [!TIP]
+> Watch [our “last mile” webinar](https://www.youtube.com/watch?v=n5gFBQTClxc&t=696s) where we describe how we used this chart to optimize Observable’s API traffic.
+
+## Implementation
+
+```
+.
+├── README.md
+├── observablehq.config.js
+├── package.json
+└── src
+    ├── components
+    │   ├── apiHeatmap.js
+    │   └── apiHistogram.js
+    ├── data
+    │   ├── latency-heatmap-avatar.parquet
+    │   ├── latency-heatmap-documents-at.parquet
+    │   ├── latency-heatmap-documents-public.parquet
+    │   ├── latency-heatmap.parquet
+    │   ├── latency-histogram.parquet
+    │   ├── size-heatmap.parquet
+    │   ├── top-routes-count.parquet
+    │   └── top-routes-duration.parquet
+    └── index.md
+```
+
+No dependencies other than Framework. No required configuration (static data). The project config only adds our example header and example layout.
+
+### Data
+
+No data loaders; static data committed to git. The data is stored as Apache Parquet files. We use `FileAttachment.parquet` to read the files in the client. The files were generated using Snowflake data loaders, but we aren’t sharing the source code for the data loaders at this time. You can see the other Snowflake data loader example.
+
+The data is represented as TKTKTK.
+
+### Components
+
+Two custom components.
 
 This example has two reusable components for building the visualizations: `apiHeatmap.js` and `apiBars.js`. Both use a combination of [Observable Plot](https://observablehq.com/plot/) and [D3](https://d3js.org/) to create highly detailed and interactive charts.
 
+Needs more explanation of how these work.

examples/api/observablehq.config.js

@@ -1,8 +1,11 @@
 export default {
   root: "src",
-  title: "Analyzing web logs",
+
+  // Shared Observable example configuration; feel free to remove this.
+  title: "Observable Framework",
   pager: false,
   toc: false,
+  sidebar: false,
   head:
     process.env.CI &&
     `<script type="module" async src="https://events.observablehq.com/client.js?pageLoad"></script>

examples/api/package.json

@@ -1,12 +1,18 @@
 {
   "type": "module",
+  "private": true,
   "scripts": {
+    "clean": "rimraf src/.observablehq/cache",
+    "build": "rimraf dist && observable build",
     "dev": "observable preview",
-    "build": "rm -rf dist && observable build",
-    "postinstall": "ln -sf ../../../../dist/bin/observable.js node_modules/.bin/observable"
+    "deploy": "observable deploy",
+    "observable": "observable"
   },
   "dependencies": {
-    "@observablehq/framework": "link:../.."
+    "@observablehq/framework": "^1.7.0"
+  },
+  "devDependencies": {
+    "rimraf": "^5.0.5"
   },
   "engines": {
     "node": ">=18"

examples/api/src/.gitignore

@@ -0,0 +1 @@
+/.observablehq/cache/