Skip to content

New Example: Monitoring Data Pipelines#230

Merged
dcodesdev merged 276 commits intoshuttle-hq:mainfrom
AdepojuJeremy:main
Sep 25, 2025
Merged

New Example: Monitoring Data Pipelines#230
dcodesdev merged 276 commits intoshuttle-hq:mainfrom
AdepojuJeremy:main

Conversation

@AdepojuJeremy
Copy link
Copy Markdown
Contributor

@AdepojuJeremy AdepojuJeremy commented Sep 19, 2025

Description

This PR adds a new Axum Template example showcasing a high-performance ETL pipeline built with Polars and OpenTelemetry observability with Axum shuttle template.

Example Details

  • Repository: https://github.com/AdepojuJeremy/polars-otel-shuttle
  • Description: Demonstrates building observable data pipelines using Polars for high-performance data processing with integrated OpenTelemetry tracing, metrics and axum shuttle templates.
  • Tech Stack: Rust, Polars, OpenTelemetry, Shuttle

Features

  • High-performance data transformations with Polars
  • Distributed tracing with OpenTelemetry
  • Performance monitoring and observability
  • One-click deployment on Shuttle

The example provides a complete reference implementation for teams looking to build production-ready data pipelines with observability built-in.

oddgrd and others added 30 commits February 6, 2023 16:03
…le-hq#31)

...folder is missing

Co-authored-by: Iulian Barbu <iul@maple.finance>
* feat: initial commit of shuttle-next example

* feat: add post endpoint

* feat: convert all examples to bins

* feat: remove syncwrapper

* feat: refactor axum hello-world

* feat: from impl for AxumService

* feat: update all axum examples

* feat: update actix examples (postgres not working)

* feat: update poem examples

* feat: update poise example

* feat: update rocket examples

* feat: update serenity and salvo

* feat: update thruster examples

* feat: update the tower and warp examples

* feat: update tide examples

* misc: bump versions (shuttle-hq#28)

* misc: v0.12.0-rc1 (shuttle-hq#29)

* refactor: update next example (shuttle-hq#30)

* chore: v0.12.0 (shuttle-hq#32)

---------

Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>
* feat: workspace example

* fix: bump shuttle versions

* chore: v0.14.0

---------

Co-authored-by: Oddbjørn Grødem <29732646+oddgrd@users.noreply.github.com>
…ample (shuttle-hq#38)

* feat examples: add rocket static files and dynamic template example projects.

* feat examples: remove unneeded static-files rocket example

* examples docs: fix incorrect comment

* feat examples: code now formatted with rustfmt and comment regarding serving static content
* feat: changing '/hello' routes to '/'

* feat: change "/hello" routes to "/"

---------

Co-authored-by: Paul Otten <potten@my.bcit.ca>
@AdepojuJeremy AdepojuJeremy changed the title Add Observable Data Pipeline Monitoring with Polars and Open Telemetry Add Axum Template: Add Observable Data Pipeline Monitoring with Polars and Open Telemetry(Polars-Otel-Shuttle) Sep 19, 2025
@dcodesdev dcodesdev changed the title Add Axum Template: Add Observable Data Pipeline Monitoring with Polars and Open Telemetry(Polars-Otel-Shuttle) New Example: Monitoring Data Pipelines Sep 23, 2025


Merge remote-tracking branch 'origin/main' into main

Reconciling divergent histories between local and remote branches.

Local changes:
- Added polars-otel-shuttle entry to community_templates section in templates.toml
- Applied cargo fmt and clippy fixes to axum/polars-otel-shuttle project files

Remote changes:
- Updates from shuttle-hq/shuttle-examples upstream repository
- Maintainer merges and repository synchronization

This merge preserves both the local template configuration changes
and the upstream repository updates to maintain PR compatibility.
templates.toml Outdated
author = "wiseaidev"
repo = "https://github.com/wiseaidev/rocket-yew-starter-pack"

[community_templates.polars-otel-shuttle]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate, remove.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted, working on them.

templates.toml Outdated

[templates.axum-polars-otel-shuttle]
title = "Polars ETL with OpenTelemetry"
description = "An Axum + Polars ETL pipeline instrumented with OpenTelemetry, deployed on Shuttle."
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

description too long, make it shorter

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is not needed, remove.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this needed?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure there are no unused dependencies here.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make this concise, it's too long.

- Fix clippy warnings (unused variables, dead code) to resolve CI failures
- Remove duplicate template entry from templates.toml
- Shorten template description
- Remove unnecessary .gitignore and .ignore files
- Clean up unused dependencies in Cargo.toml
- Make README more concise and focused
Concise readme
@dcodesdev dcodesdev merged commit 7ffe8c4 into shuttle-hq:main Sep 25, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.