diff --git a/Sources/onboarding-brewery-solution b/Sources/onboarding-brewery-solution index d66bf74..39bdad5 160000 --- a/Sources/onboarding-brewery-solution +++ b/Sources/onboarding-brewery-solution @@ -1 +1 @@ -Subproject commit d66bf7493b7b567d1c7da17256f700ace7ca5e10 +Subproject commit 39bdad5560a8181e2e7f39731743c4725860d9dd diff --git a/Tutorial/docs/tutorials/web-app/.nav.yml b/Tutorial/docs/tutorials/web-app/.nav.yml index 622b6c0..2c1b78d 100644 --- a/Tutorial/docs/tutorials/web-app/.nav.yml +++ b/Tutorial/docs/tutorials/web-app/.nav.yml @@ -1,5 +1,10 @@ nav: - index.md - Frontend: frontend.md - - ADX database: adx-database.md - - Power BI: power-bi.md + - ADX: + - Overview: adx-database.md + - Functions & Performance: adx-functions-performance.md + - Power BI: + - Overview: power-bi.md + - Parameters & Reuse: power-bi-parameters.md + - Embedding & Security: power-bi-embedding.md diff --git a/Tutorial/docs/tutorials/web-app/adx-database.md b/Tutorial/docs/tutorials/web-app/adx-database.md index 156f9fa..31aec07 100644 --- a/Tutorial/docs/tutorials/web-app/adx-database.md +++ b/Tutorial/docs/tutorials/web-app/adx-database.md @@ -1,3 +1,203 @@ -# ADX database +--- +title: ADX Database Overview +summary: Integrate and manage the Azure Data Explorer (ADX) layer for your web application. +tags: + - adx + - data + - web-app +--- -Integrate and manage the ADX database for your web application. +# ADX Database Overview + +This page introduces how the web-app consumes data from Azure Data Explorer (ADX) and how to structure your data estate for analytics & dashboards. + +## 1. Why ADX for the Solution? + +- Sub-second exploratory analytics on time-series & events +- Native integration with Power BI (DirectQuery / import) and Azure ecosystem +- Scalable ingestion & retention policies + +## 2. High-Level Flow (Project-specific) + +``` +Sources (onboarding data / storage) + └─> ETL (Cosmo Tech runner templates) + - code/run_templates/etl_with_local_file + - code/run_templates/etl_with_azure_storage + └─> Backend pipelines (Solution_etl.yaml, Create.kql) + - Apply ADX tables/mappings/policies + - Register update policies / materialized views + - Create serving functions +→ ADX Raw / Prepared Tables +→ Serving Functions +→ Power BI (embedded via backend) +``` + +Notes: +- The backend is the source of truth for ADX artifacts via `Create.kql` and CI/CD. +- Onboarding data is optional and used only in non-prod to validate end-to-end. + +## 3. Core Concepts (in this project) + +| Concept | Where it’s defined | Example | +|---------|---------------------|---------| +| Tables & Mappings | Backend `Create.kql` | `BrewerEvents`, JSON/CSV mappings | +| Update Policy / MV | Backend `Create.kql` | Raw -> Cleaned / 15-min aggregates | +| Serving Function | Backend KQL scripts | `Dashboard_Fact(...)` | + +Avoid manual table/function creation in shared envs—propose changes via PR to backend scripts. + +## 4. Schema Source of Truth & Design Patterns + +`Create.kql` dictates the schema. Use these patterns when proposing changes: + +| Column Type | Purpose | Naming | +|-------------|---------|--------| +| Timestamp | Query time axis | `Timestamp` first column | +| Entity Keys | Filter dimensions | `Site`, `BatchId`, `Product` | +| Measures | Numeric metrics | `DurationSeconds`, `Temperature` | +| Status / Type | Categorical attributes | `EventType` | + +Retention example (applied by backend): + +```kusto +.alter table BrewerEvents policy retention softdelete = 30d +``` + +## 5. Ingestion Tips (Project-specific) + +- For dev/onboarding data, use the provided ETL templates to generate the source dataset: + - Local file template: `Sources/onboarding-brewery-solution/code/run_templates/etl_with_local_file/etl.py` + - Azure Storage template: `Sources/onboarding-brewery-solution/code/run_templates/etl_with_azure_storage/etl.py` + - Sample inputs live in `Sources/onboarding-brewery-solution/Simulation/Resource/scenariorun-data/` and, when zipped, under `reference/Nodes/*.csv` and `reference/Edges/*.csv`. +- ADX ingestion is executed by the backend pipeline using `Create.kql` mappings. Do not ingest directly into curated tables. If you need to validate a schema change, ingest into a staging table defined in `Create.kql` and let update policies/materialized views populate the serving tables. +- Keep file formats aligned with the repository (CSV for Nodes/ and Edges/). For larger volumes in higher environments, consider Parquet in storage and update the ingestion mapping in `Create.kql` accordingly. +- For near real-time flows, rely on backend-managed connectors and orchestration defined under `Sources/onboarding-brewery-solution/API/` (e.g., `Solution_etl.yaml`) rather than ad-hoc ingestion; coordinate with backend to register any new sources. + +## 6. Schema Provisioning via Backend + +In this project, the ADX schema (tables, mappings, policies) is defined in the backend `Create.kql` script and deployed through backend pipelines. Do not recreate tables manually from the UI. + +Workflow for changes: +1. Propose updates in `Create.kql` (new columns, policies, mappings) +2. Run local validation against a dev cluster (optional) +3. Open a PR; backend CI applies changes to non-prod +4. After approval, pipeline promotes to production + +Keep serving function contracts stable; if breaking, version functions (e.g., `Dashboard_Fact_v2`). + +## 7. Serving Functions + +All dashboard/API queries should route via serving functions (see: [ADX Functions & Performance](./adx-functions-performance.md)). + +Benefits: +- Centralize filter logic +- Enforce parameter contracts +- Reduce duplication across Power BI & services + +## 8. Governance & Source Control + +- Store Kusto artifacts in backend repo alongside `Create.kql`. +- Use PRs and CI to apply changes to dev first, then promote. + +## 9. Observability + +Use built-in commands: +```kusto +.show table BrewerEvents details +.show functions +.show queries | where StartedOn > ago(15m) +.show operations | where StartedOn > ago(1h) and Status != 'Completed' +// If using managed ingestion into tables, check ingestion state +.show ingestion failures | where Database == '' and Table == 'BrewerEvents' and Timestamp > ago(1h) +``` + +## 10. Next + +Proceed to implementing reusable functions and performance tuning. + +> See: [Functions & Performance](./adx-functions-performance.md) + +## 11. Junior Onboarding: ADX Setup (Step-by-step) + +This walkthrough assumes: +- You have access to the Azure subscription and target ADX cluster/database +- Backend owns deployment of schema via `Create.kql` +- You have the onboarding repo checked out at `Sources/onboarding-brewery-solution` + +### A. Prerequisites (local) + +- Azure CLI (logged in): `az login` +- ADX permissions: can view database and run read-only queries +- Coordinates from your team: + - Cluster URI: `https://..kusto.windows.net` + - Database name: `` + - Backend pipeline path for `Create.kql` (read-only) + +### B. Provision (done by Backend) + +Backend CI will apply `Create.kql` to provision tables, mappings, policies and serving functions. As a junior, you don’t run schema changes yourself. + +What you should do: verify that objects exist after deployment. + +- In Kusto Explorer or Web UI, run: +```kusto +.show tables +.show functions +``` +Confirm presence of domain tables (e.g., `BrewerEvents`) and serving functions (e.g., `Dashboard_Fact` or similar per project naming). + +### C. Load Sample Data (optional, non-prod) + +If the environment is empty and your team allows sample data loads, use the onboarding dataset. + +Location in repo: +- `Sources/onboarding-brewery-solution/Simulation/Resource/scenariorun-data/` + - `Bar.csv`, `Customer.csv`, `arc_to_Customer.csv` (example graph data) + +Typical approaches (confirm which is permitted): +- Use backend ETL runner templates under `code/run_templates/` (e.g., `etl_with_local_file`) — usually interacts with Cosmo Tech API & datasets +- Or a one-off ADX ingestion (dev only) via Web UI upload to a staging table that matches `Create.kql` schemas + +Ask your mentor which path to use in your environment. + +### D. Sanity Queries (read-only) + +Run simple queries to check data landed and functions work: +```kusto +// Replace names with your actual tables/functions +BrewerEvents | take 5 +BrewerEvents | summarize count() by Site +Dashboard_Fact(datetime(ago(12h)), datetime(now()), 'Lyon_01', '') | take 10 +``` + +Expect a few rows and reasonable counts. If empty: +- Confirm ingestion happened +- Check RLS/filters (if querying through Power BI) + +### E. Power BI Integration Check + +This project embeds Power BI via backend. Your tasks: +- Confirm the dataset’s main query references the serving function(s) +- Validate that parameter values (Site, TimeWindowHours) match what the function expects +- Open the embedded dashboard and verify visuals populate for a known site/time window + +If visuals are blank, coordinate with backend to check RLS mapping and embed token scopes. + +### F. Change Requests + +When you need schema or serving function changes: +1. Discuss with backend owner +2. Propose edits in `Create.kql` or function `.kql` files +3. Open a PR → CI applies to dev → validate → promote + +Avoid ad-hoc manual changes in shared environments. + +### G. What to Document During Onboarding + +- Cluster/Database you used +- Tables/functions you validated +- Screenshots of the sanity queries results +- Any discrepancies found & who you contacted + +This helps the next newcomer follow the same path. diff --git a/Tutorial/docs/tutorials/web-app/adx-functions-performance.md b/Tutorial/docs/tutorials/web-app/adx-functions-performance.md new file mode 100644 index 0000000..e9621ef --- /dev/null +++ b/Tutorial/docs/tutorials/web-app/adx-functions-performance.md @@ -0,0 +1,177 @@ +--- +title: ADX Functions & Performance +summary: Create reusable functions and optimize Kusto queries powering the web-app dashboards. +tags: + - adx + - performance + - web-app + - optimization +--- + +# ADX Functions & Performance + +This guide shows how to encapsulate logic into Azure Data Explorer (ADX) functions and optimize query performance for responsive dashboards. + +> Focus areas: +> 1. Reuse logic across dashboards & web-app deployments +> 2. Reduce data scanned & transferred to Power BI +> 3. Improve freshness vs cost balance +> 4. Make parameters first-class citizens + +## 1. Organize Your ADX Layer + +| Layer | Purpose | Artifacts | +|-------|---------|-----------| +| Raw ingestion | Land data as-is | Ingestion mappings, staging tables | +| Harmonized | Cleaned, typed, conformed | Update policies, materialized views | +| Serving | Query-ready, thin, parameterizable | Functions, export tables | + +Keep functions in a dedicated folder in source control (e.g. `infrastructure/adx/functions/`). Version them and deploy declaratively (ARM/Bicep/Terraform / Kusto scripts). + +## 2. Defining Functions + +Two main kinds: +- Inline (lightweight) functions +- Stored functions created once in the database + +Example: `GetBrewEvents(startTime:datetime, endTime:datetime, site:string)` + +```kusto +.create-or-alter function with (docstring = 'Brew events filtered by time & site', folder='serving/brew') +GetBrewEvents(startTime:datetime, endTime:datetime, site:string) +{ + BrewerEvents + | where Timestamp between (startTime .. endTime) + | where Site == site + | project Timestamp, Site, BatchId, EventType, DurationSeconds +} +``` + +### Best Practices +- Use PascalCase for function names, camelCase for parameters. +- Provide `docstring` and `folder` for discoverability. +- Avoid hard-coded constants; expose as parameters. + +## 3. Composing Functions + +Create small focused functions and compose: + +```kusto +// Base filter +.create-or-alter function GetBatches(startTime:datetime, endTime:datetime){ + Batches | where StartTime between (startTime .. endTime) + | project BatchId, Product, StartTime, EndTime +} + +// Duration enrichment +.create-or-alter function GetBatchDurations(startTime:datetime, endTime:datetime){ + GetBatches(startTime, endTime) + | extend DurationMinutes = datetime_diff('minute', EndTime, StartTime) +} + +// Aggregated KPI +.create-or-alter function GetOutputKPI(startTime:datetime, endTime:datetime){ + GetBatchDurations(startTime, endTime) + | summarize AvgDuration = avg(DurationMinutes), TotalBatches = count() +} +``` + +## 4. Parameter Patterns for Power BI + +When embedding, the web-app can compute a time window & site, then pass them as query parameters. + +Pattern: a single root function that accepts all Power BI-relevant parameters and outputs a narrow dataset. + +```kusto +.create-or-alter function Dashboard_Fact(startTime:datetime, endTime:datetime, site:string, product:string){ + GetBrewEvents(startTime, endTime, site) + | where product == '' or Product == product + | summarize Events=count(), TotalDuration=sum(DurationSeconds) +} +``` + +In Power BI M query (parameterized): + +```m +let + StartTime = DateTimeZone.UtcNow() - #duration(0,12,0,0), + EndTime = DateTimeZone.UtcNow(), + Site = Text.From(EnvSiteParameter), + Product = "" , + Source = Kusto.Contents("https://..kusto.windows.net", "", + "Dashboard_Fact(datetime({" & DateTimeZone.ToText(StartTime, "yyyy-MM-dd HH:mm:ss") & "}), datetime({" & DateTimeZone.ToText(EndTime, "yyyy-MM-dd HH:mm:ss") & "}), '" & Site & "', '" & Product & "')") +in + Source +``` + +(Replace the dynamic site & product with actual PBI parameters linked to slicers.) + +## 5. Performance Tactics + +| Goal | Tactic | Notes | +|------|--------|-------| +| Min scan | Use narrower projection early | Always `project` right after filters | +| Fast filter | Use ingestion time partitioning & `between` | Align queries to partition keys | +| Reuse | Materialized views | For heavy joins/expensive lookups | +| Lower transfer | Summarize before export to PBI | Return aggregated rows, not raw events | +| Adaptive freshness | Tiered functions (raw vs MV) | Switch via a boolean parameter | + +### Example: Hybrid Live + Historical + +```kusto +.create-or-alter function Dashboard_Fact_Live(startAgo:timespan, site:string){ + GetBrewEvents(now()-startAgo, now(), site) +} + +.create-or-alter function Dashboard_Fact_History(startTime:datetime, endTime:datetime, site:string){ + Materialized_BrewAgg + | where WindowStart between (startTime .. endTime) + | where Site == site +} + +.create-or-alter function Dashboard_Fact_Union(startTime:datetime, endTime:datetime, liveAgo:timespan, site:string){ + union isfuzzy=true + (Dashboard_Fact_History(startTime, endTime, site)) + (Dashboard_Fact_Live(liveAgo, site)) + | summarize Events=sum(Events), TotalDuration=sum(TotalDuration) +} +``` + +## 6. Monitoring & Diagnostics + +```kusto +.show functions +.show commands-and-queries | where StartedOn > ago(1h) +``` + +Enable Query Store & alerts for: +- Long-running queries > 15s +- High data scanned vs returned ratio + +## 7. Deployment Automation + +In this project, functions are deployed by the backend CI using the authoritative Kusto scripts. Prefer adding/altering function definitions in the backend repository and let the pipeline apply them idempotently. + +If you need local validation, run them against a dev database from your admin tools, but do not hand-deploy to shared environments. + +### CI Hints + +- Store each function in its own `.kql` file with `create-or-alter` +- Group in folders by domain (e.g., `serving/brew`) +- Add lightweight unit checks (syntax validation) in CI +- Tag releases with a migration note + +## 8. Checklist + +- [ ] All dashboard queries go through a root serving function +- [ ] Parameters documented (name, type, default) +- [ ] Functions have docstrings & folder metadata +- [ ] Query completion p50 < 3s +- [ ] No function returns > 100k rows to Power BI +- [ ] Materialized views used for joins > 10M rows + +## 9. Next Steps + +Continue with Power BI parameterization to bind these functions to dynamic dashboards. + +> See: [Power BI Parameters & Reuse](./power-bi-parameters.md) diff --git a/Tutorial/docs/tutorials/web-app/frontend.md b/Tutorial/docs/tutorials/web-app/frontend.md index 57b1eec..ec5d963 100644 --- a/Tutorial/docs/tutorials/web-app/frontend.md +++ b/Tutorial/docs/tutorials/web-app/frontend.md @@ -1,3 +1,107 @@ -# Frontend +--- +title: Frontend Architecture & Integration +summary: Learn about the frontend architecture and customization of your web application. +tags: + - frontend + - web-app + - power-bi + - adx +--- -Learn about the frontend architecture and customization of your web application. +# Frontend Architecture & Integration + +The frontend orchestrates context (site, user, time window), passes parameters to embedded analytics, and optimizes perceived performance. + +## 1. Layers + +| Layer | Responsibility | Tech | +|-------|----------------|------| +| Routing | Page/screen selection | React Router / framework router | +| State | Global context (site, auth, feature flags) | Redux / Zustand / Context API | +| Services | REST / GraphQL / ADX proxy calls | Fetch / Axios | +| Embedding | Power BI iframe + SDK | `powerbi-client` | +| Components | UI widgets / filters / charts | Design system | + +## 2. Parameter Propagation Flow + +``` +User selects Site -> Global State update -> ADX function queries (server) & Embed Filters (client) -> Report visuals refresh +``` + +Core rule: single source of truth for each parameter (avoid duplicate local state). + +## 3. Environment Configuration + +Expose environment via `.env` or config endpoint: + +| Variable | Purpose | +|----------|---------| +| `PBI_AUTH_MODE` | `service-principal` / `user` | +| `PBI_BACKEND_TOKEN_URL` | Token fetch endpoint | +| `ADX_CLUSTER_URL` | Display / debugging | +| `DEFAULT_SITE` | Initial filter | + +## 4. Fetching Embed Token + +Note: Embedding is handled in the backend. The frontend typically calls a backend endpoint to retrieve the embed configuration; SDK handling may be abstracted already by platform components. + +```ts +async function fetchEmbedToken(reportId: string) { + const res = await fetch(`/api/powerbi/token?reportId=${reportId}` , { credentials: 'include' }); + if (!res.ok) throw new Error('Token failed'); + return res.json(); // { token, embedUrl, datasetId } +} +``` + +## 5. Context Synchronization + +```ts +const [site, setSite] = useState(initialSite); +useEffect(() => { + reportRef.current?.setFilters([ + { $schema: 'http://powerbi.com/product/schema#basic', target: { table: 'SiteFilter', column: 'Site' }, operator: 'In', values: [site] } + ]); +}, [site]); +``` + +## 6. Performance Optimizations + +| Concern | Mitigation | +|---------|------------| +| Slow initial dashboard | Lazy-load Power BI SDK & use skeleton loaders | +| Excess re-renders | Memoize filter components & avoid prop drilling | +| Network chatter | Debounce user input before triggering queries | +| Large bundle | Code-split embedding feature | + +## 7. Error Handling Patterns + +```ts +try { + const token = await fetchEmbedToken(reportId); + embedReport(container, { ...token, site }); +} catch (e) { + logError(e); + showToast('Unable to load analytics'); +} +``` + +Surface fallback UI with actionable steps (retry / contact support). + +## 8. Observability + +- Custom performance marks (e.g. `analytics_loaded`) +- Track embed token latency +- Capture front-end errors (Sentry / App Insights) + +## 9. Security Hygiene + +- Never store secrets in frontend bundle +- Validate all query parameter values server-side +- Sanitize user-entered strings passed to functions + +## 10. Next + +Proceed to ADX or Power BI deep dives: + +- [ADX Overview](./adx-database.md) +- [Power BI Overview](./power-bi.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi-embedding.md b/Tutorial/docs/tutorials/web-app/power-bi-embedding.md new file mode 100644 index 0000000..b96828a --- /dev/null +++ b/Tutorial/docs/tutorials/web-app/power-bi-embedding.md @@ -0,0 +1,105 @@ +--- +title: Power BI Embedding & Security +summary: Back-end managed embedding — configure workspaces, datasets, and RLS; validate filters and performance. +tags: + - power-bi + - embedding + - security + - web-app +--- + +# Power BI Embedding & Security (Backend-managed) + +In this solution, embedding is implemented in the backend. You do not write frontend embedding code. Instead, you configure the embedding context and validate security and performance. + +## 0. Onboarding Flow Snapshot + +``` +Clone repos → Verify permissions (variables.yaml / gen_adx_permissions.sh) → ADX bootstrap (00-Initialization.kql via backend) → Publish Example.pbix → Backend embed endpoint test → Open web-app → Validate RLS & filters +``` + +## 1. What’s handled by the backend (Recap) + +- Acquire AAD tokens (service principal or user) +- Generate Power BI embed tokens for the selected report/dataset +- Apply identity mappings (optional) for RLS +- Expose a simplified API endpoint to the web-app + +Your frontend only consumes the embed configuration and applies high-level report filters (e.g., site) when needed. + +## 2. Required Configuration (From `variables.yaml`, `workspace.yaml`, `solution.yaml`) + +| Setting / Element | Source File | YAML Path | Notes | +|-------------------|-------------|-----------|-------| +| Power BI Workspace Name | `variables.yaml` | `powerbi_workspace_name` | Target workspace for Example report | +| Report binding | `workspace.yaml` | `sidecars.azure.powerbi.workspace.reports` | Ensures Example.pbix deployed with parameters | +| Report parameters (ADX cluster/db) | `workspace.yaml` | `reports[].parameters` | Provide `ADX_Cluster`, `ADX_Database` values | +| ADX init script | `workspace.yaml` | `sidecars.azure.adx.scripts` | Executes `00-Initialization.kql` | +| Event Hub connectors | `workspace.yaml` | `sidecars.azure.eventhub.connectors` | Stream simulation outputs into ADX tables | +| Dynamic Filters | `workspace.yaml` | `webApp.options.charts.scenarioView.dynamicFilters` | Auto apply `LastSimulationRun` | +| Run Templates | `solution.yaml` | `spec.runTemplates` | Provide simulation & ETL context for data feeding report | + +If a principal is missing, update `variables.yaml` in a PR and re-run provisioning scripts. + +## 3. Permission Bootstrap + +Use (if provided) `gen_adx_permissions.sh` and backend automation to ensure ADX and Power BI roles are applied. Confirm in ADX: +```kusto +.show database principals +``` +And in Power BI workspace (Admin portal) that the service principal has Admin rights. + +## 4. RLS & Identity Mapping (Dataset) + +If the Example PBIX includes RLS roles (e.g., SiteIsolation), verify them in Desktop: Modeling > Manage Roles. If not present yet, coordinate with analytics engineer to add roles BEFORE wide distribution. + +Recommended role pattern: +- `TenantIsolation` (filters organization/workspace id if multi-tenant) +- `ScenarioScope` (limits to allowed scenario set) + +Backend maps user/app context → role(s) when generating embed token (or injects effective identity). + +## 5. Validation Steps + +1. Publish `Example.pbix` to workspace (`powerbi_workspace_name`). +2. Call embed endpoint (e.g., `GET /api/analytics/embed/report/{id}`) and capture JSON (id, embedUrl, token expiry). +3. Open web-app dashboard; verify: + - Data loads without manual sign-in (service principal path) + - Only allowed scenarios/probes appear + - Switching TimeWindowHours (if parameterized) re-queries ADX +4. In ADX, list recent queries: +```kusto +.show queries | where StartedOn > ago(10m) | where Text has "GetMeasures" or Text has "GetScenarios" +``` +Confirm calls originate with expected principal. +5. Measure render time (<5s for main visuals) using browser dev tools. + +## 6. Troubleshooting (Extended) + +| Symptom | Likely Cause | Diagnostic | Fix | +|---------|--------------|-----------|-----| +| Empty visuals | RLS over-filtering | View as role in Desktop | Adjust role filter / identity mapping | +| 403 embed call | Missing workspace permission | Power BI workspace access panel | Add service principal / user | +| Slow queries | Function expanding large dynamic fields | Inspect ADX query text | Add selective projection / MV | +| Token expires too soon | Short TTL | Backend token config | Increase within security guidelines | +| Wrong dataset | Misconfigured IDs | Compare variables.yaml vs embed response | Correct backend config | + +## 7. Performance (Applied) + +- Monitor `GetMeasures` query duration distribution (p50 < 2s target in dev) using `.show queries`. +- If dynamic expansion (bag_unpack) dominates cost, pre-flatten heavy fields in a materialized view and reference that in functions. +- Schedule dataset refresh only if Import mode; for DirectQuery ensure aggregations are defined before scaling. + +## 8. Operational Runbook (Daily / Release) + +| Task | Frequency | Owner | +|------|-----------|-------| +| Check failed embed calls (5xx / 4xx) | Daily | Backend Ops | +| Review ADX heavy queries (>10s) | Daily | Data Eng | +| Validate RLS after adding new users | On change | Platform Admin | +| Refresh Example PBIX (if Import) | Release / Data change | BI Dev | + +## 9. Related + +- [Parameters & Reuse](./power-bi-parameters.md) +- [ADX Functions & Performance](./adx-functions-performance.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi-parameters.md b/Tutorial/docs/tutorials/web-app/power-bi-parameters.md new file mode 100644 index 0000000..ffa4885 --- /dev/null +++ b/Tutorial/docs/tutorials/web-app/power-bi-parameters.md @@ -0,0 +1,115 @@ +--- +title: Power BI Parameters & Reuse +summary: Design parameterized datasets and M queries to reuse dashboards across environments and web-app deployments. +tags: + - power-bi + - parameters + - reuse + - performance +--- + +# Power BI Parameters & Reuse + +Parametrization lets you deploy the same `.pbix` to multiple tenants, sites, or time scopes without cloning logic. It also reduces dataset size by pushing filters down into ADX. + +## 1. Parameter Strategy (Project Context) + +In this training setup, parameters align with serving functions created in `00-Initialization.kql` (e.g., `GetScenarios`, `GetMeasures`). Rather than many separate filters, we focus on a small stable set: + +| Parameter | Purpose | Backing Function Usage | Source | +|-----------|---------|------------------------|--------| +| Site (or Workspace scope) | Narrow scenario / probe data | Filters Scenario/Probe tables (if multi-site) | Derived from workspace / user claim | +| TimeWindowHours | Limit recency for probe facts | Compute datetime range in Kusto (`ago(TimeWindowHours * 1h)`) | Frontend control | +| ProbeType | Focus a specific measure set | Passed to `GetMeasures(Probe)` | Report slicer | +| ScenarioState (optional) | Filter scenarios by validation state | Filter in `GetScenarios()` pipeline | Report slicer | + +Keep parameters minimal; they must map cleanly to function parameters or filters without post-processing. + +### Mapping to Configuration Files + +| Concept | Declared In | How It Appears In Report | +|---------|-------------|--------------------------| +| Run template parameters (e.g., CustomersCount) | `project/solution.yaml` (`parameters` & `parameterGroups`) | May influence simulation outputs surfaced via `GetScenarios()` | +| Dynamic filter (last run) | `project/workspace.yaml` (`dynamicFilters`) | Automatically filters report to latest simulation run (no manual parameter) | +| ADX connection parameters | `workspace.yaml` powerbi.workspace.reports.parameters (`ADX_Cluster`, `ADX_Database`) | M parameters bound during deployment | +| Probe / scenario functions | `00-Initialization.kql` | Data sources in Power BI queries | + +## 2. Creating Parameters in Power BI Desktop (Applied) + +Use only parameters required for query shaping. Example minimal set: `TimeWindowHours`. +If Site is handled via RLS or embedding context, skip exposing it as a manual parameter. + +Steps: +1. Transform Data > Manage Parameters > New > `TimeWindowHours` (Number, default 12) +2. (Optional) Add `ProbeType` as Text if you want a configurable default instead of a slicer. +3. If cluster/database differ per environment, add `ADXCluster` & `ADXDatabase` parameters; populate from deployment script or Power BI deployment pipeline rules. + +## 3. Using Parameters in Kusto M Query (GetMeasures) + +```m +let + Cluster = Parameter_ADXCluster, + Database = Parameter_ADXDatabase, + Hours = Number.ToText(Parameter_TimeWindowHours), + // Example: restrict to last N hours then expand probes + Query = + "GetMeasures('LatencyProbe') | where ProbeDate > ago(" & Hours & "h)" +, + Source = Kusto.Contents(Cluster, Database, Query) +in + Source +``` + +For scenarios list: +```m +let + Cluster = Parameter_ADXCluster, + Database = Parameter_ADXDatabase, + Query = "GetScenarios()" +, + Source = Kusto.Contents(Cluster, Database, Query) +in + Source +``` + +## 4. Dynamic Binding (Embedding Context) + +In this platform the backend embedding already injects user/workspace context. Frontend may only adjust slicers (e.g., ProbeType). Avoid duplicating Site/Workspace filters if RLS or function logic already enforces them. + +## 5. Dataset Slimming (Project-Specific) + +| Technique | Application Here | Notes | +|-----------|------------------|-------| +| Function parameters | Limit probe window | Use `ago()` in function or query | +| Pre-aggregation | Add MV on `ProbesMeasures` if performance degrades | Requires backend PR | +| Column pruning | Remove unused dynamic fields | Avoid expanding entire `FactsRaw` unless needed | +| Incremental refresh | If converting to Import mode for large history | Partition by `ProbeDate` | + +## 6. Testing Reusability + +Instead of exporting PBIX per site, validate across workspace contexts (if multi-workspace deployment) and time windows: + +```bash +for hours in 6 12 24; do + echo "Testing TimeWindowHours=$hours" + # Use pbicli or REST to rebind parameter (pseudo-example) + pbicli datasets update-params --dataset-id $DATASET --parameters TimeWindowHours=$hours || exit 1 + sleep 5 + # Optionally trigger a refresh if Import mode + echo "OK" +done +``` + +## 7. Checklist (Adapted) + +- [ ] Queries call only approved serving functions (GetMeasures/GetScenarios) +- [ ] Parameter list minimal & documented +- [ ] No duplicated filter logic in visuals and functions +- [ ] Dataset refresh / DirectQuery latency within targets +- [ ] Performance test across TimeWindowHours values + +## 8. Next + +Proceed to backend embedding & security validation. + +> See: [Embedding & Security](./power-bi-embedding.md) diff --git a/Tutorial/docs/tutorials/web-app/power-bi.md b/Tutorial/docs/tutorials/web-app/power-bi.md index 85241c3..dba577e 100644 --- a/Tutorial/docs/tutorials/web-app/power-bi.md +++ b/Tutorial/docs/tutorials/web-app/power-bi.md @@ -1,3 +1,102 @@ -# Power BI +--- +title: Power BI Overview +summary: Embed and use Power BI dashboards in your web application. +tags: + - power-bi + - web-app +--- -Embed and use Power BI dashboards in your web application. +# Power BI Overview + +This section introduces how Power BI integrates with the solution for interactive analytics. + +## 1. Objectives + +- Reuse the same report across environments & sites +- Keep dashboards performant (< 5s load for key visuals) +- Enforce data security & isolation + +## 2. Architecture Snapshot + +``` +ADX (Functions) -> Power BI Dataset -> Report -> Embedded in Web-App +``` + +## 3. Key Building Blocks (Extended) + +| Component | Source File | Purpose | +|-----------|------------|---------| +| Solution metadata | `project/solution.yaml` | Declares run templates (Example, ETL) & parameters feeding scenarios | +| Workspace definition | `project/workspace.yaml` | Defines Power BI workspace, ADX scripts, Event Hub connectors, webApp embedding options | +| ADX Init Script | `project/adx/scripts/00-Initialization.kql` | Creates tables & functions consumed by report | +| Power BI Report | `project/powerbi/Example.pbix` | Visualization layer referencing serving functions | +| Permissions | `variables.yaml` (`powerbi_permissions`, `adx_permissions`) | Grants principals access | +| Dynamic Filters | `workspace.yaml` (`webApp.options.charts.scenarioView.dynamicFilters`) | Auto-apply context filters (e.g., last simulation run) | + +## 4. Quick Start Checklist (Onboarding Flow) + +Follow these in order when onboarding: + +1. Clone repos: + - `onboarding-brewery-solution` (simulation + ETL templates) + - `build-my-app-training-workspaces` (workspace/Power BI + ADX bootstrap scripts) +2. Review `build-my-app-training-workspaces/variables.yaml` and note: + - `organization_id`, `workspace_key`, `powerbi_workspace_name` + - Power BI & ADX principal entries (`powerbi_permissions`, `adx_permissions`) +3. Ensure your user (or service principal) has been added under `powerbi_permissions` and `adx_permissions` (if not, submit a PR or request an admin update and re-run provisioning scripts). +4. Run ADX initialization (if dev environment is fresh): + - Execute `project/adx/scripts/00-Initialization.kql` via the approved backend pipeline or Kusto Script deployment job (never hand-edit prod). +5. Import / verify `Example.pbix` into the Power BI workspace named in `variables.yaml` if not already published. (Admins only; otherwise just confirm it's present.) +6. Open the report and validate required dataset parameters (Site, TimeWindowHours, etc.) exist or are represented via filters / serving functions (e.g., `GetMeasures`, `GetScenarios`). +7. Confirm backend embedding endpoint returns an embed token for the report (use internal API doc / curl if available). +8. Load the web-app; verify visuals render for default site and switch site/time context successfully. +9. Capture performance: measure first visual load (<5s target) & check ADX query durations for `GetMeasures` and `GetScenarios` functions. +10. Log findings in onboarding notes (successes, missing permissions, performance outliers). + +## 5. Key Building Blocks + +| Component | Repository / File | Purpose | +|-----------|------------------|---------| +| Workspace Variables | `build-my-app-training-workspaces/variables.yaml` | IDs, permissions, workspace naming | +| ADX Bootstrap | `project/adx/scripts/00-Initialization.kql` | Tables, mappings, baseline functions | +| Example Report | `project/powerbi/Example.pbix` | Starter visuals tied to functions | +| Serving Functions | Defined in KQL bootstrap | `GetScenarios`, `GetMeasures`, etc. | +| ETL Templates | `onboarding-brewery-solution/code/run_templates/*` | Populate dataset / scenario data | + +## 6. Performance Tips (Contextual) + +| Issue | Mitigation | +|-------|------------| +| Slow `GetMeasures` | Pre-aggregate or reduce `mv-expand` usage | +| Large dataset refresh | Limit time window in parameters (TimeWindowHours) | +| Permission errors | Re-check `variables.yaml` and re-run permission scripts | + +## 7. Next Steps + +Continue with parameter configuration: see [Parameters & Reuse](./power-bi-parameters.md) + +> After parameters and embedding validation, proceed to RLS & operational governance in [Embedding & Security](./power-bi-embedding.md) + +## 8. Unified Onboarding Checklist + +| Step | Action | File / Command | Done | +|------|--------|----------------|------| +| 1 | Clone repos | git clone both repos | [ ] | +| 2 | Inspect variables | Open `variables.yaml` | [ ] | +| 3 | Confirm PB workspace name | `powerbi_workspace_name` in `variables.yaml` | [ ] | +| 4 | Check solution run templates | `solution.yaml` runTemplates section | [ ] | +| 5 | Review workspace report binding | `workspace.yaml` powerbi.workspace.reports | [ ] | +| 6 | Verify ADX scripts referenced | `workspace.yaml` adx.scripts + `00-Initialization.kql` | [ ] | +| 7 | Ensure principals present | `powerbi_permissions` / `adx_permissions` | [ ] | +| 8 | Provision / sync backend | Run pipeline / deployment process | [ ] | +| 9 | Publish or confirm `Example.pbix` | Power BI Service | [ ] | +| 10 | Test embed endpoint | API call (embed token) | [ ] | +| 11 | Open web-app & validate visuals | Browser | [ ] | +| 12 | Run ADX sanity queries | `.show tables`, `GetScenarios()` | [ ] | +| 13 | Check dynamic filter (last run) | `workspace.yaml` dynamicFilters config | [ ] | +| 14 | Performance sanity | Measure load & ADX query p50 | [ ] | +| 15 | Document findings | Onboarding notes | [ ] | + +## 9. Next Steps + +Proceed to parameter specifics: [Parameters & Reuse](./power-bi-parameters.md) and embedding governance: [Embedding & Security](./power-bi-embedding.md)