Merge pull request #47969 from TheJamesHerring/DP-700-Ignite-orchestrate-processes-in-fabric

DP700-Ignite-new-module-

Author: JillGrant615

learn-pr/wwl/orchestrate-processes-in-fabric/1-introduction.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.orchestrate-processes-in-fabric.introduction
+title: Introduction
+metadata:
+  title: Introduction
+  description: "Introduction"
+  ms.date: 11/14/2024
+  author: jamesh
+  ms.author: jamesh
+  ms.topic: unit
+  ms.custom:
+  - DP-700
+durationInMinutes: 3
+content: |
+  [!include[](includes/1-introduction.md)]

learn-pr/wwl/orchestrate-processes-in-fabric/2-choose-between-pipeline-notebook.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.orchestrate-processes-in-fabric.choose-between-pipeline-notebook
+title: Describe Microsoft Fabric Real-Time Intelligence?
+metadata:
+  title: Describe Microsoft Fabric Real-Time Intelligence?
+  description: "What is Microsoft Fabric Real-Time Intelligence?"
+  ms.date: 11/14/2024
+  author: jamesh
+  ms.author: jamesh
+  ms.topic: unit
+  ms.custom:
+  - DP-700
+durationInMinutes: 10
+content: |
+  [!include[](includes/2-choose-between-pipeline-notebook.md)]

learn-pr/wwl/orchestrate-processes-in-fabric/3-design-schedules-and-event-based-triggers.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.orchestrate-processes-in-fabric.design-schedules-and-event-based-triggers
+title: Design schedules and event based triggers
+metadata:
+  title: Design schedules and event based triggers
+  description: "Design pipeline schedules and event based triggers."
+  ms.date: 11/14/2024
+  author: jamesh
+  ms.author: jamesh
+  ms.topic: unit
+  ms.custom:
+  - DP-700
+durationInMinutes: 15
+content: |
+  [!include[](includes/3-design-schedules-and-event-based-triggers.md)]

learn-pr/wwl/orchestrate-processes-in-fabric/4-exercise-implement-dynamic-practices-to-notebooks.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.orchestrate-processes-in-fabric.exercise-implement-dynamic-patterns-to-notebooks
+title: Implement and schedule a dynamic notebook in a fabric pipeline
+metadata:
+  title: Implement and schedule a dynamic notebook in a fabric pipeline
+  description: "Implement and schedule a dynamic notebook in a fabric pipeline."
+  ms.date: 11/14/2024
+  author: jamesh
+  ms.author: jamesh
+  ms.topic: unit
+  ms.custom:
+    - DP-700
+durationInMinutes: 15
+content: |
+  [!include[](includes/4-exercise-implement-dynamic-patterns-to-notebooks.md)]

learn-pr/wwl/orchestrate-processes-in-fabric/5-knowledge-check.yml

@@ -0,0 +1,49 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.orchestrate-processes-in-fabric.knowledge-check
+title: Knowledge Check
+metadata:
+  title: Knowledge Check
+  description: "Knowledge check"
+  ms.date: 11/14/2024
+  author: jamesh
+  ms.author: jamesh
+  ms.topic: unit
+  ms.custom:
+  - DP-700
+durationInMinutes: 3
+quiz:
+  title: ""
+  questions:
+  - content: "What is a key feature of Dataflows in Microsoft Fabric?"
+    choices:
+    - content: "Dataflows only support data ingestion from a limited number of sources."
+      isCorrect: false
+      explanation: "Incorrect. Dataflows can ingest data from hundreds of sources."
+    - content: "Dataflows offer a low-code interface to ingest and transform data from numerous sources."
+      isCorrect: true
+      explanation: "Correct. Dataflows provide a low-code interface for data ingestion from various sources and transformation using over 300 data transformations."
+    - content: "Dataflows can't be run manually or on a schedule."
+      isCorrect: false
+      explanation: "Incorrect. Dataflows can be run manually, on a schedule, or as part of a data pipeline."
+  - content: "What is the purpose of a storage event trigger in Fabric Data Factory pipelines?"
+    choices:
+    - content: "To automate data pipelines based on events occurring in storage accounts."
+      isCorrect: true
+      explanation: "Correct. Storage event triggers in Fabric Data Factory pipelines are used to automate data pipelines when specific events occur in storage accounts."
+    - content: "To manually start a data pipeline run."
+      isCorrect: false
+      explanation: "Incorrect. Storage event triggers are used to automate data pipelines based on events, not for manual initiation."
+    - content: "To schedule regular data pipeline runs."
+      isCorrect: false
+      explanation: "Incorrect. While scheduling is a feature of Fabric Data Factory, it isn't the function of storage event triggers."
+  - content: "What is required to build an Eventhouse that contains a KQL database?"
+    choices:
+    - content: "A Microsoft Fabric trial license with the Fabric preview enabled in your tenant."
+      isCorrect: true
+      explanation: "Correct. A Microsoft Fabric trial license with the Fabric preview enabled in your tenant is required to build an Eventhouse that contains a KQL database."
+    - content: "A Microsoft Office 365 subscription."
+      isCorrect: false
+      explanation: "Incorrect. A Microsoft Office 365 subscription alone isn't sufficient to build an Eventhouse with a KQL database."
+    - content: "A Google Cloud trial license with the Fabric previews enabled."
+      isCorrect: false
+      explanation: "Incorrect. Google Cloud trial license isn't required for building an Eventhouse with a KQL database."

learn-pr/wwl/orchestrate-processes-in-fabric/6-summary.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.orchestrate-processes-in-fabric.summary
+title: Summary
+metadata:
+  title: Summary
+  description: "Summary"
+  ms.date: 11/14/2024
+  author: jamesh
+  ms.author: jamesh
+  ms.topic: unit
+  ms.custom:
+  - DP-700
+durationInMinutes: 1
+content: |
+  [!include[](includes/6-summary.md)]

learn-pr/wwl/orchestrate-processes-in-fabric/includes/1-introduction.md

@@ -0,0 +1,22 @@
+This module covers various units that provide an overview of some of the data orchestration and data movement technologies within Microsoft Fabric. It provides an overview of the scheduling and event-based triggering capabilities, which can help data engineers design pro-active systems to support the rapidly changing data needs of business.
+
+## Pipelines in Microsoft Fabric
+
+Data Factory provides a modern data integration experience to ingest, prepare, and transform data from various sources like databases, data warehouses, Lakehouse, and real-time data. It supports both citizen and professional developers with intelligent transformations and a rich set of activities. Users can create pipelines to execute multiple activities, access data sources through linked services, and add triggers to automate processes. Data Factory in Microsoft Fabric introduces Fast Copy capabilities for rapid data movement between data stores, enabling efficient data transfer to Lakehouse and Data Warehouse for analytics.
+
+## Notebooks in Microsoft Fabric
+
+Notebooks in Microsoft Fabric offer a versatile environment for data exploration, transformation, and analysis. They support various programming languages, including Python, KQL, and SQL, and provide an interactive interface for running code, visualizing data, and documenting workflows. Key features include:
+
+## Schedules and Triggers in Microsoft Fabric
+
+Scheduling pipelines includes three standard types of scheduling, including time-based, event-based, and custom scheduling. Scheduling in Microsoft Fabric also allows for seamless integration with other services, which allow for more dynamic event-driven pipelines.
+
+The articles covered in this module include:
+
+- Understanding pipelines in Microsoft Fabric.
+- Understanding notebooks in Microsoft Fabric.
+- Event-based triggers and scheduling in Microsoft Fabric.
+- practice some dynamic features of Microsoft Fabric notebooks
+
+Students learn how Data Factory in Microsoft Fabric is used for modern data integration, including ingesting, preparing, and transforming data from various sources. They understand how to create pipelines, automate processes with triggers, and utilize Fast Copy for rapid data movement to Lakehouse and Data Warehouse. Additionally, they explore Notebooks in Microsoft Fabric for interactive data exploration, multi-language support, data visualization, and collaboration. The training provides an overview of integration with other Microsoft Fabric services, event-based triggers, scheduling, and dynamic features of notebooks, providing an understanding of data workflows within Microsoft Fabric.

learn-pr/wwl/orchestrate-processes-in-fabric/includes/2-choose-between-pipeline-notebook.md

@@ -0,0 +1,63 @@
+
+## Data Factory Pipelines in Microsoft Fabric
+
+Data Factory provides a modern way to integrate data, allowing you to collect, prepare, and transform data from various sources like databases, data warehouses, Lakehouse, real-time data, and more. Whether you're a beginner or an experienced developer, you can use intelligent transformations and a wide range of activities to process your data.
+
+### Dataflows overview
+
+Dataflows offer a low-code interface to ingest data from numerous sources and transform it using over 300 data transformations. The transformed data can be loaded into various destinations, such as Azure SQL databases. Dataflows can be run manually, on a schedule, or as part of a data pipeline.
+
+### Important features of Dataflows
+
+- **Low-Code Interface**: Ingest data from hundreds of sources.
+- **Transformations**: Utilize 300+ data transformations.
+- **Destinations**: Load data into multiple destinations like Azure SQL databases.
+- **Execution**: Run dataflows manually, on a schedule, or within a data pipeline.
+
+### Power Query integration
+
+Dataflows are built using the Power Query experience, available across Microsoft products like Excel, Power BI, and Power Platform. Power Query enables users, from beginners to professionals, to perform data ingestion and transformations with ease. It supports joins, aggregations, data cleansing, custom transformations, and more, all through a user-friendly, visual, low-code interface.
+
+### Real-World Uses Cases for Dataflows
+
+**Data Consolidation for Reporting**:
+Organizations often have data spread across multiple sources such as databases, cloud storage, and on-premises systems. Dataflows can be used to consolidate this
+data into a single, unified dataset, which can then be used for reporting and analytics. For example, a company might use Dataflows to combine sales data from different regions into a single dataset for a comprehensive sales report. This single dataset can be further curated and promoted into a semantic model for use by a larger audience.
+
+**Data Preparation for Machine Learning**:
+Dataflows can be used to prepare and clean data for machine learning models. This method includes tasks such as data cleansing, transformation, and feature engineering. For instance, a data science team might use Dataflows to preprocess customer data, removing duplicates and normalizing values before feeding it into a machine learning model.
+
+**Real-Time Data Processing**:
+Dataflows can handle real-time data ingestion and transformation, making them ideal for scenarios where timely data processing is crucial. For example, an e-commerce platform might use Dataflows to process real-time transaction data, updating inventory levels and generating real-time sales reports.
+
+**Data Migration**:
+When migrating data from legacy systems to modern platforms, Dataflows can be used to extract, transform, and load (ETL) data into the new system. This process ensures that data is accurately and efficiently transferred, minimizing downtime and data loss. For instance, a company migrating from an on-premises database to Azure SQL Database might use Dataflows to handle the data migration process.
+
+**Self-Service Data Preparation**:
+Dataflows provide a low-code interface that allows business users to prepare their own data without needing extensive technical knowledge. This approach empowers users to create their own dataflows for tasks such as data cleansing, transformation, and enrichment, reducing the dependency on IT teams. For example, a marketing team might use Dataflows to prepare campaign data for analysis.
+
+These use cases demonstrate the flexibility and power of Dataflows in handling various data integration and transformation task and show a powerful self-service feature. Self-service might be more appealing to your organization's business users while still providing a roadmap to a larger ELT project that utilizes pipelines and notebooks.
+
+### Data Pipelines
+
+Data pipelines offer powerful workflow capabilities at cloud-scale, enabling you to build complex workflows that can refresh your dataflow, move petabyte-sized data, and define sophisticated control flow pipelines.
+
+### Important features of data Pipelines
+
+- **Complex Workflows**: Build workflows that can refresh dataflows, move large volumes of data, and define control flow pipelines.
+- **ETL and Data Factory Workflows**: Create complex ETL (Extract, Transform, Load) and data factory workflows which perform various tasks at scale.
+- **Control Flow Capabilities**: Utilize built-in control flow features to build workflow logic with loops and conditionals.
+
+### End-to-End ETL Data Pipeline
+
+Combine a configuration-driven copy activity with your low-code dataflow refresh in a single pipeline for a complete ETL data pipeline. You can also add code-first activities for Spark Notebooks, SQL scripts, stored procedures, and more.
+
+## Notebooks in Microsoft Fabric
+
+- **Interactive Data Exploration:** Notebooks allow users to interactively explore and analyze data, making it easier to understand and manipulate datasets.
+- **Multi-language Support:** Users can write and execute code in multiple languages within the same notebook, enhancing flexibility and collaboration.
+- **Visualization:** Notebooks support rich data visualization, enabling users to create charts, graphs, and other visual representations of data.
+- **Collaboration:** Notebooks facilitate collaboration by allowing multiple users to work on the same document simultaneously, share insights, and track changes.
+- **Integration with Fabric Services:** Notebooks seamlessly integrate with other Microsoft Fabric services, such as Data Factory, Synapse Data Engineering, and Synapse Data Science. This approach provides a unified platform for end-to-end data workflows.
+
+When comparing these technologies, it's important to note that while Data Factory focuses on data integration and pipeline automation, notebooks in Microsoft Fabric provide an interactive and ***collaborative*** environment for data exploration, documentation, transformation, and analysis. Both tools complement each other, offering a comprehensive solution for managing and analyzing data within the Microsoft Fabric ecosystem.

learn-pr/wwl/orchestrate-processes-in-fabric/includes/3-design-schedules-and-event-based-triggers.md

@@ -0,0 +1,76 @@
+## Scheduling and event-based triggers
+
+Within Fabric, you have several options that include both traditional and proactive in nature to schedule jobs. Most are even-driven and we see with items, such as pipeline runs which have on-demand and scheduled ***runs***. The more proactive ***runs*** are triggered by an event, such as a file arriving in a location and can allow your data and information to be processed based on your business needs and events rather than having to wait for the next schedule to run.
+
+## Pipeline runs
+
+A **data pipeline run** happens when you start a data pipeline. This action means all the tasks in the pipeline are carried out until they're finished. 
+
+For example, if you have a task to **copy data**, running the pipeline performs that task and copy your data. Each time you run a pipeline, it gets a special identifier called a **pipeline run ID**.
+
+You can start a data pipeline in two ways:
+
+1. **On-demand**: You manually trigger it whenever you need it.
+2. **Scheduled**: You set it to run automatically at specific times and frequencies that you choose.
+
+### On-demand pipeline runs
+
+Just as you would expect, on-demand, or ad-hoc runs happen by browsing to the specific pipeline and selecting the **Run** button. You're prompted to save your changes, but the pipeline receives a ***pipeline run ID*** and you can then view the status of the ***run*** by selecting the ***Output*** tab.
+
+[![Screenshot of pipeline runs with on-demand action and their output.](../media/pipeline-run-output-jobs.png)](../media/pipeline-run-output-jobs-expanded.png#lightbox)
+
+### Scheduling pipeline runs
+
+When you schedule a data pipeline run, you can choose the frequency at which your pipeline operates.
+
+1. **Select Schedule**: 
+   - This option is found in the top banner of the **Home** tab to view your scheduling options.
+
+    [![Screenshot of pipeline runs scheduling button to build schedule.](../media/pipeline-scheduling.png)](../media/pipeline-scheduling-expanded.png#lightbox)
+
+1. **Default Setting**:
+   - By default, your data pipeline has no schedule defined.
+   - Select the **On** radio button under the **Scheduled Run** header
+
+    [![screenshot of pane for setting pipeline schedule settings.](../media/pipeline-schedule-settings.png)](../media/pipeline-schedule-settings-expanded.png#lightbox)
+
+1. **Schedule Configuration**:
+   - On the **Schedule configuration page**, you can specify:
+     - **Schedule frequency**
+     - **Start and end dates and times**
+     - **Time zone**
+2. **Apply Your Schedule**:
+   - Once you configure your settings, select **Apply** to set your schedule.
+3. **Editing Your Schedule**:
+   - You can view or edit the schedule at any time by selecting the **Schedule** button again.
+
+## Storage Event Triggers in Fabric Data Factory pipelines
+
+Storage event triggers are a powerful feature in Data Factory pipelines that allow you to automate your data pipelines based on events occurring in your storage accounts.
+
+## What Are Storage Event Triggers?
+
+One of the most common scenarios for using event triggers is to activate a data pipeline when:
+
+- **A file arrives**:  This event action means a new file is added to your storage.
+- **A file is deleted**: This event occurrence indicates a file is removed from your storage.
+
+[![Image of pane for setting storage event trigger.](../media/storage-event-trigger.png)](../media/storage-event-trigger-expanded.png#lightbox)
+
+For users moving from Azure Data Factory (ADF) to Microsoft Fabric, it's common to work with events from Azure Data Lake Storage (ADLS) or Blob storage. If you're new to Fabric and ADF, you might be more familiar with file events from **OneLake**.
+
+## How Triggers Work in Fabric
+
+In Fabric Data Factory, triggers utilize advanced platform features, including:
+
+- **Event Streams**: These triggers allow you to listen for specific events in real-time.
+- **Reflex Triggers**: These triggers are designed to respond quickly to events.
+
+
+### Creating a Trigger
+
+To create a trigger in the Fabric Data Factory:
+
+1. Open the **pipeline design canvas**.
+2. Look for the **Trigger** button, which allows you to create a Reflex trigger for your pipeline.
+3. Alternatively, you can create triggers directly from the [**Data Activator**](/fabric/data-activator/data-activator-get-started) experience.

learn-pr/wwl/orchestrate-processes-in-fabric/includes/4-exercise-implement-dynamic-patterns-to-notebooks.md

@@ -0,0 +1,9 @@
+Now it's your chance to build and schedule a pipeline with a dynamic notebook in a pipeline activity.
+<!-- This link should still be valid after, but may need revisited prior to BUILD -->
+
+> [!NOTE]
+> You need a Microsoft Fabric trial license with the Fabric preview enabled in your tenant. See [**Getting started with Fabric**](/fabric/get-started/fabric-trial) to enable your Fabric trial license.
+
+Launch the exercise and follow the instructions.
+
+[![Icon for Button to launch exercise.](../media/launch-exercise.png)](https://go.microsoft.com/fwlink/?linkid=2260721)