Skip to content

Commit 74c07f6

Browse files
authored
add a project-level README file (#2)
* add a project-level README file * add readme * minor improvements
1 parent 26f4323 commit 74c07f6

File tree

1 file changed

+205
-0
lines changed

1 file changed

+205
-0
lines changed

README.md

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
# Smart Factory Monitoring App
2+
3+
This project showcases a comprehensive smart factory monitoring application specifically designed to demonstrate **LocalStack for Snowflake** capabilities for local data cloud development, debugging, and testing throughout the entire software development lifecycle (SDLC).
4+
5+
This application showcases the full spectrum of **LocalStack for Snowflake** features, such as:
6+
7+
- Emulating a local Snowflake environment with databases, schemas, tables, and more.
8+
- Utilizing S3 service for automated data ingestion with Snowpipe integration.
9+
- Implementing dbt transformations for data quality and processing.
10+
- Building interactive dashboards with Streamlit and Snowflake Native Apps.
11+
- Orchestrating data transformations in an automated pipeline with Dagster.
12+
- Implementing comprehensive data & integration testing with pytest and dbt tests.
13+
- Using GitHub Actions for continuous integration to ensure automated testing.
14+
- Adding real-time alerting with a Python User-Defined Function (UDFs) and Snowflake Tasks.
15+
- Showcasing Zero-Copy Cloning for instant data recovery scenarios in case of data loss.
16+
17+
The application serves as a complete reference implementation for developing Snowflake data solutions locally, enabling real-time monitoring of factory machines, automated data ingestion, quality testing, and predictive maintenance alerts—all running entirely on your local machine through LocalStack's Snowflake emulator.
18+
19+
## Architecture
20+
21+
The application implements a modern data pipeline architecture showcasing a realistic smart factory scenario:
22+
23+
```
24+
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
25+
│ Data Sources │───▶│ S3 Bucket │───▶│ Snowpipe │
26+
│ (CSV Sensors) │ │ (LocalStack) │ │ (Auto-Ingest) │
27+
└─────────────────┘ └──────────────┘ └─────────────────┘
28+
29+
30+
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
31+
│ Streamlit │◀───│ dbt │◀───│ Snowflake │
32+
│ Dashboard │ │ Transformations│ │ Emulator │
33+
└─────────────────┘ └──────────────┘ └─────────────────┘
34+
│ │ │
35+
▼ ▼ ▼
36+
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
37+
│ Native App │ │ Dagster │ │ Email Alerts │
38+
│ (Snowflake) │ │ Pipeline │ │ (SES) │
39+
└─────────────────┘ └──────────────┘ └─────────────────┘
40+
```
41+
42+
## Prerequisites
43+
44+
Features in this sample app require a LocalStack for Snowflake license - make sure your Auth Token is configured in your terminal session.
45+
46+
```bash
47+
export LOCALSTACK_AUTH_TOKEN=<your-token>
48+
```
49+
50+
- [Docker](https://docs.docker.com/get-docker/)
51+
- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli).
52+
- [AWS CLI](https://docs.localstack.cloud/user-guide/integrations/aws-cli/) with the [`awslocal` wrapper](https://docs.localstack.cloud/user-guide/integrations/aws-cli/#localstack-aws-cli-awslocal).
53+
- [Snowflake CLI](https://docs.snowflake.com/developer-guide/snowflake-cli/index) with a [`localstack` connection profile](https://snowflake.localstack.cloud/user-guide/integrations/snow-cli/).
54+
- [Python](https://www.python.org/downloads/) 3.10+ & [`pip`](https://pip.pypa.io/en/stable/installation/)
55+
- [`make`](https://www.gnu.org/software/make/) (**optional**, but recommended for running the sample application)
56+
57+
## Installation
58+
59+
To run the sample application, you need to install the required dependencies.
60+
61+
First, clone the repository:
62+
63+
```bash
64+
git clone https://github.com/localstack-samples/snowflake-smart-factory-app.git
65+
```
66+
67+
Then, navigate to the project directory:
68+
69+
```bash
70+
cd snowflake-smart-factory-app
71+
```
72+
73+
Next, install the project dependencies by running the following command:
74+
75+
```bash
76+
make install
77+
```
78+
79+
## Deployment
80+
81+
Start LocalStack with the `LOCALSTACK_AUTH_TOKEN` pre-configured:
82+
83+
```bash
84+
localstack auth set-token <your-auth-token>
85+
DOCKER_FLAGS='-e SF_LOG=trace' \
86+
DEBUG=1 \
87+
IMAGE_NAME=localstack/snowflake:latest \
88+
localstack start
89+
```
90+
91+
To deploy the sample application, run the following command:
92+
93+
```bash
94+
make deploy
95+
```
96+
97+
This will:
98+
99+
- Setup Snowflake database, tables, and Snowpipe
100+
- Setup S3 bucket with event notifications
101+
- Upload sample sensor data (`data/sensor_data_batch_1.csv`)
102+
- Run dbt transformations to process the data
103+
104+
You can also run the following command to deploy the Native app:
105+
106+
```bash
107+
make app
108+
```
109+
110+
The output will be similar to the following:
111+
112+
```bash
113+
App is now accessible at https://snowflake.localhost.localstack.cloud:4566/apps/test/test/FACTORY_APP_HARSHCASPER/
114+
```
115+
116+
The dashboard provides:
117+
118+
- Real-time machine health monitoring
119+
- Interactive sensor data visualizations
120+
- Predictive maintenance recommendations
121+
- Anomaly detection and alerting
122+
123+
## Testing
124+
125+
You can run full end-to-end integration tests using the following command:
126+
127+
```bash
128+
make test
129+
```
130+
131+
This executes:
132+
133+
- **Machine Health Tests**: Verifies health metrics calculations and thresholds
134+
- **Snowpipe Integration Tests**: Tests automated data ingestion workflows
135+
136+
## Use Cases
137+
138+
### Pipeline Orchestration
139+
140+
To run automated dbt transformations on new data, you can launch a Dagster pipeline:
141+
142+
```bash
143+
make pipeline
144+
```
145+
146+
This will:
147+
148+
- Setup a pipeline monitoring dashboard
149+
- Create a S3 sensor to monitor new data
150+
- Trigger dbt transformations on new data
151+
152+
To trigger dbt transformations on new data, you can run the following command:
153+
154+
```bash
155+
make upload LATEST=true
156+
```
157+
158+
This will upload the latest sensor data batch (`data/sensor_data_batch_2.csv`) to the S3 bucket. The Dagster pipeline will automatically detect the new data and trigger the dbt transformations.
159+
160+
To generate new sensor data, you can run the following command:
161+
162+
```bash
163+
make generate
164+
```
165+
166+
### Real-Time Alerting
167+
168+
To set up automated email alerts for critical machine conditions, you can run the following command:
169+
170+
```bash
171+
make alerts
172+
```
173+
174+
This will:
175+
176+
- Create a Python UDF to send email alerts with SES integration
177+
- Create a Snowflake Task to trigger the UDF every 30 seconds
178+
- Create a Snowflake View to query critical machine data
179+
- Send HTML emails for critical machine conditions
180+
181+
In 30 seconds, you can query the SES developer endpoint to see the alert email:
182+
183+
```bash
184+
curl -s http://localhost.localstack.cloud:4566/_aws/ses
185+
```
186+
187+
You can also use the [Mailhog extension](https://github.com/localstack/localstack-extensions/tree/main/mailhog) to view the alert email via a user-friendly interface.
188+
189+
### Data Recovery with Zero-Copy Cloning
190+
191+
To demonstrate instant data recovery capabilities, you can run the following command:
192+
193+
```bash
194+
snow sql -f solutions/data_recovery_clones.sql -c localstack
195+
```
196+
197+
This showcases:
198+
199+
- Instant table cloning without data duplication
200+
- Point-in-time recovery scenarios
201+
- Disaster recovery best practices
202+
203+
## License
204+
205+
This project is licensed under the [Apache License 2.0](LICENSE).

0 commit comments

Comments
 (0)