Skip to content

Commit c6aa235

Browse files
brown9804brown9804
authored
+ HowMonitorChanges.md
1 parent 1b43254 commit c6aa235

File tree

1 file changed

+131
-0
lines changed

1 file changed

+131
-0
lines changed

‎Workloads-Specific/DataFactory/HowMonitorChanges.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Demostration: How to Track Pipeline Modifications
2+
3+
Costa Rica
4+
5+
[![GitHub](https://img.shields.io/badge/--181717?logo=github&logoColor=ffffff)](https://github.com/)
6+
[brown9804](https://github.com/brown9804)
7+
8+
Last updated: 2025-03-03
9+
10+
----------
11+
12+
<details>
13+
<summary><b>List of References </b> (Click to expand)</summary>
14+
15+
- [What is Azure Data Factory?](https://learn.microsoft.com/en-us/azure/data-factory/introduction)
16+
- [Quickstart: Get started with Azure Data Factory](https://learn.microsoft.com/en-us/azure/data-factory/quickstart-get-started)
17+
- [Quickstart: Create a data factory by using the Azure portal](https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory)
18+
19+
</details>
20+
21+
<details>
22+
<summary><b>Table of Content </b> (Click to expand)</summary>
23+
24+
- [How to create a Data Factory in Azure](#how-to-create-a-data-factory-in-azure)
25+
- [Create a pipeline](#create-a-pipeline)
26+
- [How to see who modified a pipeline](#how-to-see-who-modified-a-pipeline)
27+
28+
</details>
29+
30+
## How to create a Data Factory in Azure
31+
32+
1. **Log in to Azure Portal**: Open your web browser and go to the Azure Portal. Enter your credentials to log in.
33+
2. **Search for Data Factory**: Use the search bar at the top to search for `Data Factory` and select `Data Factory` from the results.
34+
35+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/78b857da-1550-41f4-80bd-ea81fdafc24c" />
36+
37+
3. **Create a New Data Factory**:
38+
- Click on the `+ Create` button.
39+
- In the "Basics" tab, fill in the required fields:
40+
- **Subscription**: Select your Azure subscription.
41+
- **Resource Group**: Select an existing resource group or create a new one.
42+
- **Region**: Choose the region where you want to deploy the Data Factory.
43+
- **Name**: Enter a unique name for your Data Factory.
44+
- **Version**: Select V2 (the latest version).
45+
4. **Configure Git (Optional)**: If you want to configure Git for source control, you can do so in the `Git configuration` tab. This step is optional and can be skipped if not needed.
46+
47+
> [!NOTE]
48+
> Or later (crucial for source control or auditing):
49+
<img width="250" alt="image" src="https://github.com/user-attachments/assets/e5dfe437-6d76-4b51-9b87-5fb82d455f15">
50+
51+
5. **Review and Create**:
52+
- Click on the `Review + create` button.
53+
- Review your settings and click `Create` once the validation passes.
54+
55+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/3dbdc4a4-2a41-487e-9a2a-628921381dfe" />
56+
57+
6. **Wait for Deployment**: The deployment process will take a few minutes. Once it's complete, you will see a notification.
58+
7. **Access Data Factory**: After the deployment is complete, click on the `Go to resource` button to access your new Data Factory.
59+
8. **Launch Data Factory Studio**: In the Data Factory resource page, click on the `Launch Studio` tile to launch the Data Factory Studio where you can start creating pipelines and other data integration tasks.
60+
61+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/2fae2d31-54d0-40f0-adab-111f5b464cab" />
62+
63+
## Create a pipeline
64+
65+
1. **Log in to Azure Portal**: Open your web browser and go to the Azure Portal. Enter your credentials to log in.
66+
2. **Go to Data Factory**: Use the search bar at the top to search for `Data Factory` and select your Data Factory instance from the list.
67+
3. **Launch Data Factory Studio**: In the Data Factory resource page, click on the `Launch Studio` tile to launch the Data Factory Studio where you can start creating pipelines and other data integration tasks.
68+
69+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/2fae2d31-54d0-40f0-adab-111f5b464cab" />
70+
71+
4. **Create a New Pipeline**:
72+
- Click on the `New` next to `Pipelines` in the tree view.
73+
- Select `Pipeline` from the dropdown menu.
74+
75+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/5f112ab1-5327-49d9-bce6-b8ee98f23267" />
76+
77+
5. **Add Activities to the Pipeline**:
78+
- In the pipeline canvas, click on the `Activities` pane on the left.
79+
- Drag and drop the desired activities (e.g., Copy Data, Data Flow) onto the pipeline canvas.
80+
81+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/fd21dfaf-8e3f-4265-8ff7-1f44098b4827" />
82+
83+
6. **Configure Activities**:
84+
- Click on each activity on the canvas to configure its properties.
85+
- For example, if you are using a Copy Data activity, you will need to specify the source and destination datasets.
86+
87+
7. **Set Up Linked Services**:
88+
- Linked services are used to define the connection information for data sources and destinations.
89+
- Go to the `Manage` tab on the left, then click on `Linked services`.
90+
- Click on the **+ New** button to create a new linked service and configure the connection details.
91+
92+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/a6ca86d0-754d-4b1d-bd27-8f45da00fbe0" />
93+
94+
8. **Create Datasets**:
95+
- Datasets represent the data structures within the data stores.
96+
- Go to the `Author` tab, then click on `Datasets`.
97+
- Click on the `+ (plus) icon` to create a new dataset and configure its properties.
98+
99+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/56ef28cb-8e57-4e70-8322-7ad6777bb886" />
100+
101+
9. **Validate the Pipeline**: Click on the `Validate` button at the top of the pipeline canvas to check for any errors or missing configurations.
102+
10. **Publish the Pipeline**: Once validation is successful, click on the `Publish All` button to save and publish your pipeline.
103+
11. **Trigger the Pipeline**: Click on `Trigger now` to run the pipeline immediately, or configure a trigger for scheduled runs.
104+
105+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/0c42c3e3-57a8-4111-add4-1ea7192b6ac8" />
106+
107+
12. **Monitor Pipeline Runs**: In the `Monitor` tab, you can view the status of pipeline runs, check for any errors, and review the execution details.
108+
109+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/11922b16-2d9e-49d8-b0ab-13605a18018f" />
110+
111+
## How to see who modified a pipeline
112+
113+
1. **Log in to Azure Portal**: Open your web browser and go to the Azure Portal. Enter your credentials to log in.
114+
2. **Go to Azure Data Factory**: Once logged in, use the search bar at the top to search for `Data Factory` and select your Data Factory instance from the list.
115+
3. **Open the Activity Log**:
116+
- In the Data Factory resource page, look for the `Activity log` option in the left-hand menu under the `Monitoring` section.
117+
- Click on `Activity log` to open the log view.
118+
4. **View Activity Log Details**:
119+
- In the Activity Log, you will see a list of events related to your Data Factory.
120+
- You can see columns such as `Operation Name`, `Status`, `Event Initiated By`, `Time`,`Subscription`, and more.
121+
5. **Filter and Search**:
122+
- Use the filters at the top to narrow down the events by time range, resource group, resource, and more.
123+
- You can also use the search bar to find specific events or operations.
124+
6. **Review Event Details**: Click on any event in the list to view more detailed information about that event, including the JSON payload with additional properties.
125+
126+
<img width="550" alt="image" src="https://github.com/user-attachments/assets/07cf4582-6b7b-451e-94e9-9557cdbfd09f">
127+
128+
<div align="center">
129+
<h3 style="color: #4CAF50;">Total Visitors</h3>
130+
<img src="https://profile-counter.glitch.me/brown9804/count.svg" alt="Visitor Count" style="border: 2px solid #4CAF50; border-radius: 5px; padding: 5px;"/>
131+
</div>

0 commit comments

Comments
 (0)