Skip to content

Commit 960a201

Browse files
committed
finish first round
1 parent 3b5d9a5 commit 960a201

File tree

86 files changed

+369
-359
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

86 files changed

+369
-359
lines changed

.vscode/settings.json

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"dbt.enableNewLineagePanel": true
3+
}

claude/terminology.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,12 @@ This guide helps maintain consistent, data-team-friendly language across all Rec
1717
| **validation results** | diff output, comparison data | What users see in Recce |
1818
| **impact analysis** | dependency analysis, lineage | Understanding downstream effects |
1919
| **data changes** | code changes, model changes | What users are validating |
20+
| **code change** | code modification | Specific term for data team workflows |
2021
| **validation workflow** | testing workflow, QA process | How teams use Recce |
2122
| **diff** | comparison, delta | Data teams familiar with git diff, use freely |
23+
| **Recce**| recce | this is the brand name, should always use "Recce" |
24+
| **dbt** | DBT | this is the brand name, should always use lower cases|
25+
2226

2327
### Data vs Software Terms
2428

@@ -27,6 +31,7 @@ This guide helps maintain consistent, data-team-friendly language across all Rec
2731
| **data warehouse** | database | Data teams distinguish warehouses from operational databases |
2832
| **development stage** | environment | "Environment" confuses (warehouse vs dev/prod) |
2933
| **data models** | components | dbt models vs software components |
34+
| **model** | node | Data teams think "model", software teams think "node" |
3035
| **release changes** | deploy | Data teams "release" changes, don't "deploy" infrastructure |
3136
| **validation checks** | unit tests | Data quality checks vs code functionality tests |
3237
| **automated validation** | CI/CD pipeline | Data processing vs deployment automation |

claude/writing-principles.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,9 +155,21 @@ These structure is to align users adoption process. Start from using in PR, coll
155155

156156

157157

158-
## Content Structure Template
158+
## Content format guide
159159

160-
Use markdown
160+
1. **Use markdown** for all content formatting
161+
2. **Add line breaks** between sentences and bullet points or numbered lists to ensure proper rendering
162+
3. **Page title capitalization**: Capitalize every word in the main page title (Title Case)
163+
4. **Heading capitalization**: For H2 headings and below, only capitalize the first word (Sentence case)
164+
5. **Image format**: Use the standardized figure format with shadow styling:
165+
166+
```markdown
167+
<figure markdown>
168+
![Recce Lineage Diff](../assets/images/3-view-modified/lineage-diff.gif){: .shadow}
169+
<figcaption>Interactive lineage graph showing modified models</figcaption>
170+
</figure>
171+
```
172+
6. Use bold only when it's neccessary. Too many highlight will lose focus and not easy to read.
161173

162174
### Page Structure Template
163175
```markdown

docs/2-getting-started/start-free-with-cloud.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ If you don't have a dbt project, you can just click "Launch" to see the Jaffle S
5959
## Step 4: Automate Everything → CI/CD Integration
6060

6161
- **What you'll unlock**: Automatic validation on every PR
62-
- **Setup needed**: [Set up automated workflows](../7-workflow-integration/automate-in-pr.md)
62+
- **Setup needed**: [Set up automated workflows](../7-cicd/scenario-ci.md)
6363
- **Ultimate value**:
6464
-**Automatic PR checks** - validation runs on every push
6565
-**Team workflows** - standardized validation across team
@@ -69,7 +69,7 @@ If you don't have a dbt project, you can just click "Launch" to see the Jaffle S
6969

7070
After launching Recce, explore:
7171

72-
- [View modified](../3-view-modified/lineage.md)
72+
- [View modified](../3-visualized-change/lineage.md)
7373
- [View downstream impacts](../4-downstream-impacts/impact-radius.md)
7474
- [Data diffing](../5-data-diffing/query.md)
7575

docs/3-view-modified/column-level-lineage.md

Lines changed: 0 additions & 26 deletions
This file was deleted.
Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,16 @@
11
---
2-
title: Code Diff
2+
title: Code Change
33
---
44

5-
# Code Diff
5+
# Code Change
66

7-
The Code Diff feature allows you to compare the SQL code changes between your current branch and the base branch, helping you understand exactly what has been modified in your dbt models.
7+
The Code Change feature allows you to compare the SQL code changes between your current branch and the base branch, helping you understand exactly what has been modified in your dbt models.
88

9-
## Viewing Code Changes
9+
## Viewing Code Change
1010

1111
When you identify a modified model in the [Lineage Diff](lineage.md), you can examine the specific code changes to understand the nature of the modifications.
1212

13-
### Opening Code Diff
13+
### Opening Code Change
1414

1515
To view the code changes for a model:
1616

@@ -19,7 +19,7 @@ To view the code changes for a model:
1919
3. The code diff will display showing the changes between branches
2020

2121
<figure markdown>
22-
![View code changes for a model](../assets/images/3-view-modified/view-code-of-model.png){: .shadow}
22+
![View code changes for a model](../assets/images/3-visualized-change/view-code-of-model.png){: .shadow}
2323
<figcaption>Viewing code changes for a modified model</figcaption>
2424
</figure>
2525

@@ -46,7 +46,7 @@ For complex changes or detailed review, you can expand the code diff to full scr
4646
3. Use this view when conducting thorough code reviews or sharing changes with team members
4747

4848
<figure markdown>
49-
![Expanded full-screen view of code changes](../assets/images/3-view-modified/full-view-code-of-model.png){: .shadow}
49+
![Expanded full-screen view of code changes](../assets/images/3-visualized-change/full-view-code-of-model.png){: .shadow}
5050
<figcaption>Full-screen view for detailed code review</figcaption>
5151
</figure>
5252

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
title: Column-Level Lineage
3+
---
4+
5+
Column-Level Lineage provides visibility into the upstream and downstream relationships of a column.
6+
7+
Common use-cases for column-level lineage are
8+
9+
1. **Source Exploration**: During development, column-level lineage helps you understand how a column is derived.
10+
2. **Impact Analysis**: When modifying the logic of a column, column-level lineage enables you to assess the potential impact across the entire DAG.
11+
3. **Root Cause Analysis**: Column-level lineage helps identify the possible source of errors by tracing data lineage at the column level.
12+
13+
## Usage
14+
15+
1. Select a node in the lineage DAG, then click the column you want to view.
16+
17+
![alt text](../assets/images/3-visualized-change/cll-1.png){: .shadow}
18+
19+
1. The column-level lineage for the selected column will be displayed.
20+
21+
![alt text](../assets/images/3-visualized-change/cll-2.png){: .shadow}
22+
23+
1. To exit column-level lineage view, click the close button in the upper-left corner.
24+
25+
![alt text](../assets/images/3-visualized-change/cll-3.png){: .shadow}
26+
27+
# Transformation Types
28+
29+
The transformation type is also displayed for each column, which will help you understand how the column was generated or modified.
30+
31+
| Type | Description |
32+
|------|--------------|
33+
| Pass-through |The column is directly selected from the upstream table. |
34+
| Renamed | The column is selected from the upstream table but with a different name. |
35+
| Derived | The column is created through transformations applied to upstream columns, such as calculations, conditions, functions, or aggregations. |
36+
| Source | The column is not derived from any upstream data. It may originate from a seed/source node, literal value, or data generation function. |
37+
| Unknown | We have no information about the transformation type. This could be due to a parse error, or other unknown reason. |
38+
39+
Lines changed: 33 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -19,19 +19,45 @@ Data lineage tracks the flow and transformation of data through your dbt project
1919
From the Lineage view, you can determine which models to investigate further and perform various data validation checks that serve as proof-of-correctness of your work.
2020

2121
<figure markdown>
22-
![Recce Lineage Diff](../assets/images/3-view-modified/lineage-diff.gif){: .shadow}
22+
![Recce Lineage Diff](../assets/images/3-visualized-change/lineage-diff.gif){: .shadow}
2323
<figcaption>Interactive lineage graph showing modified models</figcaption>
2424
</figure>
2525

2626
!!! tip "Getting Started"
2727
When you first open Recce, the lineage graph automatically loads showing only the models affected by your changes. This focused view helps you quickly understand the impact of your work.
2828

29+
### Filter Nodes
30+
31+
In the top control bar, you can change the rule to filter the nodes:
32+
33+
1. **Mode:**
34+
- **Changed Models:** Modified nodes and their downstream + 1st degree of their parents.
35+
- **All:** Show all nodes.
36+
1. **Package:** Filter by dbt package names.
37+
1. **Select:** Select nodes by [node selection](./node-selection.md).
38+
1. **Exclude:** Exclude nodes by [node selection](./node-selection.md).
39+
40+
### Select Nodes
41+
42+
Click a node to select it, or click the **Select nodes** button in the top-right corner to select multiple nodes for further operations. For detail, see the [Multi Nodes Selections](#multi-nodes-selection) section
43+
44+
### Row Count Diff
45+
46+
A row count diff can be performed on nodes selected using the `select` and `exclude` options:
47+
48+
![](../assets/images/3-visualized-change/row-count-diff-selector.gif){: .shadow}
49+
50+
After selecting nodes, run the row count diff by:
51+
52+
1. Clicking the 3 dots (**...**) button in the top-right corner.
53+
2. Clicking **Row Count Diff by Selector**.
54+
2955
## Understanding Model Nodes
3056

3157
### Visual Status Indicators
3258

3359
<figure markdown>
34-
![Node example](../assets/images/3-view-modified/node.png){: .shadow}
60+
![Node example](../assets/images/3-visualized-change/node.png){: .shadow}
3561
<figcaption>Example model node with status indicators</figcaption>
3662
</figure>
3763

@@ -52,15 +78,15 @@ Each model node displays two icons in the bottom-right corner that indicate dete
5278
Grayed-out icons indicate no changes were detected in that category.
5379

5480
<figure markdown>
55-
![Model with Schema Change detected](../assets/images/3-view-modified/model-schema-change-detected.png){: .shadow}
81+
![Model with Schema Change detected](../assets/images/3-visualized-change/model-schema-change-detected.png){: .shadow}
5682
<figcaption>Model with Schema Change detected</figcaption>
5783
</figure>
5884

5985
!!! note "Row Count Detection"
6086
The row count icon only appears after you've run a row count diff on that specific model. This helps you track which models you've already validated.
6187

6288
<figure markdown>
63-
![Open node details panel](../assets/images/3-view-modified/node-details-panel.gif){: .shadow}
89+
![Open node details panel](../assets/images/3-visualized-change/node-details-panel.gif){: .shadow}
6490
<figcaption>Open the node details panel</figcaption>
6591
</figure>
6692

@@ -83,7 +109,7 @@ Schema diff helps you understand structural changes to your models.
83109
Click on a model to view its schema diff in the node details panel.
84110

85111
<figure markdown>
86-
![Recce Schema Diff](../assets/images/3-view-modified/schema-diff.gif){: .shadow}
112+
![Recce Schema Diff](../assets/images/3-visualized-change/schema-diff.gif){: .shadow}
87113
<figcaption>Interactive schema diff showing column changes</figcaption>
88114
</figure>
89115

@@ -97,7 +123,7 @@ Schema diff identifies:
97123
- **Data type changes**: Modifications to column types
98124

99125
<figure markdown>
100-
![Recce Schema Diff](../assets/images/3-view-modified/schema-diff.png){: .shadow}
126+
![Recce Schema Diff](../assets/images/3-visualized-change/schema-diff.png){: .shadow}
101127
<figcaption>Schema diff showing renamed column</figcaption>
102128
</figure>
103129

@@ -122,7 +148,7 @@ Learn more about viewing and analyzing code changes in the [Code Diff guide](cod
122148
The node details panel provides comprehensive information about the selected model:
123149

124150
<figure markdown>
125-
![Explore the model](../assets/images/3-view-modified/explore-the-model.png){: .shadow}
151+
![Explore the model](../assets/images/3-visualized-change/explore-the-model.png){: .shadow}
126152
<figcaption>Node details panel with exploration options</figcaption>
127153
</figure>
128154

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
---
2+
title: Multi-Models
3+
---
4+
5+
6+
## Multi-Models Selection
7+
8+
Multiple models can be selected in the Lineage DAG. This enables actions to be performed on multiple models at the same time such as Row Count Diff, or Value Diff.
9+
10+
### Select Models Individually
11+
12+
To select multiple models individually, click the checkbox on the models you wish to select.
13+
14+
<figure markdown>
15+
![Select multiple models individually](../assets/images/3-visualized-change/multi-node-selection.gif){: .shadow}
16+
<figcaption>Select multiple models individually</figcaption>
17+
</figure>
18+
19+
### Select Parent or Child models
20+
21+
To select a node and all of its parents or children:
22+
23+
1. Click the checkbox on the node.
24+
2. Right-click the node.
25+
3. Click to select either parent or child models.
26+
27+
<figure markdown>
28+
![Select a node and its parents or children](../assets/images/3-visualized-change/select-node-children.gif){: .shadow}
29+
<figcaption>Select a node and its parents or children</figcaption>
30+
</figure>
31+
32+
### Perform actions on multiple models
33+
34+
After selecting the desired models, use the Actions menu at the top right of the screen to perform diffs or add checks.
35+
36+
<figure markdown>
37+
![Perform actions on multiple models](../assets/images/3-visualized-change/actions-menu.png){: .shadow}
38+
<figcaption>Perform actions on multiple models</figcaption>
39+
</figure>
40+
41+
### Example - Row Count Diff
42+
43+
An example of selecting multiple models to perform a multi-node row count diff:
44+
45+
<figure markdown>
46+
![Perform a Row Count Diff on multiple models](../assets/images/3-visualized-change/multi-node-row-count-diff.gif){: .shadow}
47+
<figcaption>Perform a Row Count Diff on multiple models</figcaption>
48+
</figure>
49+
50+
### Example - Value Diff
51+
52+
An example of selecting multiple models to perform a multi-node Value Diff:
53+
54+
<figure markdown>
55+
![Perform a Value Diff on multiple models](../assets/images/3-visualized-change/multi-node-value-diff.gif){: .shadow}
56+
<figcaption>Perform a Value Diff on multiple models</figcaption>
57+
</figure>
58+
59+
60+
### Schema and Lineage Diff
61+
62+
From the Lineage DAG, click the Actions dropdown menu and click Lineage Diff or Schema Diff from the Add to Checklist section. This will add:
63+
64+
- Lineage Diff: The current Lineage view, dependent on your <a href="#select-models">model selection</a> options.
65+
- Schema Diff: A diff of all models if none are selected, or specific <a href="#multi-node-selection">selected models</a>.
66+
67+
<figure markdown>
68+
![Add a Lineage Diff Check or Schema Check via the Actions dropdown menu](../assets/images/3-visualized-change/actions-dropdown.png){: .shadow}
69+
<figcaption>Add a Lineage Diff Check or Schema Check via the Actions dropdown menu</figcaption>
70+
</figure>
71+
72+
73+
74+
Recce supports dbt [node selection](https://docs.getdbt.com/reference/node-selection/syntax) in the [lineage diff](./lineage.md#lineage-diff). This enables you to target specific resources with data checks by selecting or excluding models.
75+
76+
## Supported syntax and methods
77+
78+
Since Recce uses dbt's built-in node selector, it supports most of the selecting methods. Here are some examples:
79+
80+
- Select a node: `my_model`
81+
- select by tag: `tag:nightly`
82+
- Select by wildcard: `customer*`
83+
- Select by graph operators: `my_model+`, `+my_model`, `+my_model`, `1+my_model+`
84+
- Select by union: `model1 model2`
85+
- Select by intersection: `stg_invoices+,stg_accounts+`
86+
- Select by state: `state:modified`, `state:modified+`
87+
88+
89+
### Use `state` method
90+
91+
In dbt, you need to specify the `--state` option in the CLI. In Recce, we use the base environment as the state, allowing you to use the selector on the fly.
92+
93+
94+
### Removed models
95+
Another difference is that in dbt, you cannot select removed models. However, in Recce, you can select removed models and also find them using the graph operator. This is a notable distinction from dbt's node selection capabilities.
96+
97+
98+
## Supported Diff
99+
100+
In addition to lineage diff, other types of diff also support node selection. You can find these features in the **...** button in the top right corner. Currently supported diffs include:
101+
102+
- Lineage diff
103+
- Row count diff
104+
- Schema diff
105+
106+
![](../assets/images/3-visualized-change/node-selection.png){: .shadow}
107+
108+
## Limitation
109+
110+
- ["result" method](https://docs.getdbt.com/reference/node-selection/syntax#the-result-status) not supported
111+
- ["source_status" method](https://docs.getdbt.com/reference/node-selection/syntax#the-source_status-status) not supported.
112+
- [YAML selectors](https://docs.getdbt.com/reference/node-selection/yaml-selectors) not supported.

0 commit comments

Comments
 (0)