Unify on agent_memory_toolkit_v2 and add the Agent Analytics & Optimization plane - #74
Open
Mark Brown (markjbrown) wants to merge 255 commits into
Open
Unify on agent_memory_toolkit_v2 and add the Agent Analytics & Optimization plane#74Mark Brown (markjbrown) wants to merge 255 commits into
Mark Brown (markjbrown) wants to merge 255 commits into
Conversation
…olkit_v2 update readme
…rfImprovements Updating Module 01-06
…dataUpdates Updating checkpoint to use the new azure-langchain package
Branch mjbrown/unify-v2 is based on upstream/agent_memory_toolkit_v2. This commit layers on the durable, architecture-agnostic assets: - Carry forward .github/copilot-instructions.md from main (v2 lacked it). - Bring the full analytics/ initiative (Open Analytics Schema, ADRs 0001-0005, data generator/enricher, acceptance scenarios, Power BI, vision/charter). - ADR-0006: adopt v2 as the unified baseline and land v2 + dependency modernization + analytics in one PR back to main (maintainer-approved). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
v2 shipped store_debug_log/store_debug_log_from_response orphaned (never called) and OTel commented out, so the token/agent/cost analytics pillars produced nothing. Capture token usage, model metadata, tool calls, and supervisor->sub-agent delegations from v2's astream_events stream in chat_event_generator and persist one Debug log per turn. - _extract_msg_usage: native usage_metadata + response_metadata.token_usage fallback (langchain-core 1.x). - _persist_turn_debug_log: derive agent_path/handoff_count from sub-agent TOOL calls (find_places, create_or_update_itinerary) since v2 invokes sub-agents as tools, not graph nodes. - store_debug_log: additive optional agent_path/handoff_count propertyBag. - thread debug_log_id from the completion so debugLogId matches the message. Validated live on TravelAssistantV2: total_tokens 23025 (in 22211/out 814/ cached 15488), agent_path supervisor,find_places,create_or_update_itinerary, handoff_count 2. ADR-0007 records the Debug-first, OTel-later decision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Allow resuming an interrupted generation run from the Nth persona (1-based) without duplicating earlier personas' sessions/memories. run() takes a start_index so log numbering stays aligned to the full persona list. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Trips were never saved for the requesting user (Trips=0 in analytics):
the itinerary ReAct sub-agent filled user_id/tenant_id on the trip MCP
tools itself and hallucinated placeholders (e.g. user_id='user'), so
trips landed under the wrong partition; and the prompt's two-step
create_new_trip -> update_trip flow made the model echo a bad trip id
into update_trip ('Trip tripId not found').
- Inject the request (user_id, tenant_id) into create_new_trip/update_trip/
get_trip_details via a _current_identity ContextVar + _wrap_trip_tool,
mirroring the existing preference-vector injection. The sub-agent LLM can
no longer set identity.
- itinerary_agent.prompty: create a new trip in a SINGLE create_new_trip call
with the full days array (no fragile follow-up update_trip); reserve
update_trip for a real existing trip_id, used verbatim.
Validated live: trip persisted as trip_trip_user3_barcelona_... (3 days)
under the correct user; MCP log shows 'Creating trip for user: trip_user3';
no ToolException, single create call.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
v2's entry points log emoji heavily but did not reconfigure stdout, so on a default Windows console (cp1252) the first emoji log raises UnicodeEncodeError. Port the PR AzureCosmosDB#71 reconfigure block to the v2 entry points (mcp_http_server, travel_agents_api, seed_data). Verified on this machine without PYTHONUTF8: default stdout=cp1252 -> reconfigured to utf-8 -> emoji print succeeds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The classic routing eval asserts which specialist NODE ran (orchestrator->hotel/dining/activity/itinerary_generator); v2 has no such nodes. Add a v2 routing eval that tracks which sub-agent TOOL the supervisor delegated to (find_places / create_or_update_itinerary) via on_tool_start events, with membership-based correctness (itinerary fans out to find_places first). Runs fully local (no LangSmith / LANGCHAIN_API_KEY) against the 02_completed reference solution, matching the analytics initiative's Cosmos-first philosophy (ADR-0003). Validated 7/7 delegations; README documents the model and the tool-usage/e2e follow-ups. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
_mcp_recall_memories_tool was assigned filter_tools_by_prefix(...) LIST, but recall_memories_tool calls it as a single tool (.ainvoke), so every agent memory recall failed with a list-has-no-ainvoke error (the is-None guard does not catch a list). Memory creation was unaffected (SDK auto-trigger), but the agent never read stored memories to personalize. Take the first matched tool (or None). Validated: after the fix, asking maya_chen about dietary preferences recalls her stored memories (pescatarian, vegan options, peanut allergy, budget near Khao San Road); 0 recall errors in logs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…fix) Module-02 taught the itinerary sub-agent tools without identity injection, so a student's saved trips would land under a hallucinated user_id (the same bug fixed in the reference solution, commit 0603b54). Propagate the fix into both the teaching walkthrough and the Module Solution: - add `from contextvars import ContextVar` - add `_current_identity` ContextVar + `_wrap_trip_tool` / `_with_identity_injection` helpers next to the shared sub-agent helpers - wrap `_mcp_itinerary_tools` with `_with_identity_injection` - set/reset `_current_identity` around the itinerary sub-agent call in `create_or_update_itinerary_tool` - explanatory prose in Activities 1, 3, and 5 Module Solution code block verified with py_compile. LangSmith content in Modules 05/06 is untouched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ics Schema" Universal rename for clarity - "Open Analytics Schema" was too general. Updates all 20 prose occurrences across ADRs, charter, and vision; renames adr-0002 file + slug and fixes the README index link. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Establish the lab's optimization-scenario framework, anchored to the vision: - the two tiers of business questions (analytical + action) - the eight optimization dimensions (agent quality, workflow efficiency, memory effectiveness, routing, tool utilization, model selection, cost efficiency, business outcomes) - signal inventory (what TravelAssistantV2 already captures) with Trips.status as the shared outcome anchor - fix-seam safety tiers (prompt/config = safe one-click; code = human-review) - three discovery methods (data-first mining, behavioral probes, naive UI use) - catalog of 8 scenarios spanning all dimensions SCEN-001 (supervisor re-asks for a city it could infer from the active trip) is fully worked: symptom, root-cause evidence, detection metric + SQL sketch, dashboard card, prompt fix, and before/after. Behavior intentionally left unfixed as the canonical lab example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… risk domains Align the optimization-scenarios catalog with the vision's Optimization Maturity Model (L1 Visibility -> L5 Adaptive) and its human-governed risk model: prompt/workflow/code changes are HIGHER-risk (human-governed, ceiling L3 Assisted), while memory salience/retention, retrieval weighting, routing thresholds, tool/model-selection policies are LOWER-risk (autonomous-eligible, L4/L5). Corrects the earlier prompt=one-click framing; retags SCEN-001 as a prompt fix capped at L3, and marks the policy/threshold scenarios as the ones that reach self-adapting L4/L5. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…004/007 First data-first mining pass over the v2_analytics baseline (291 Debug turns, 760 memories, 11 trips), persisted as baseline-findings.md. Real numbers validate the candidates and honestly PARK SCEN-006 (token/turn is flat - no drift in short sessions). Promote the two strongest L4/L5 (autonomous) examples to full worked scenarios, complementing SCEN-001's L3 prompt example so the lab spans the maturity ladder: - SCEN-007 model selection: 48% of turns are trivial (no delegation, <60 output tokens) yet run on the full model; 86% cache already. Teaches the two-step path to autonomy (build the model-router seam once = L3 code, then tune the policy autonomously = L4/L5). - SCEN-004 stale-memory retention: 67% of memories superseded; salience populated (mean 0.82). Knobs already exist (TTL/salience/cadence) so it is autonomous-ready today - the cleanest L4/L5 example. Update the catalog: SCEN-003/005/008 marked validated; SCEN-006 parked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The data-first discovery tool behind baseline-findings.md. Reads only signal the app already captures (Debug/Messages/Trips/memories) and prints the Tier-1 metrics per candidate scenario. Verified to reproduce the documented numbers (48% trivial turns, 67% stale memories, 90% place-intent no-delegation, 219k tokens/outcome). Parameterized by --tenant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tunity, not current behavior) Make explicit that the app builds one shared model used by the supervisor and all sub-agents (100% gpt-4.1-mini in the baseline). Model selection is an opportunity dimension here - SCEN-007 introduces per-turn/per-task routing rather than tuning existing variation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… utilization) SCEN-003: the north-star cost-per-successful-outcome KPI (219k tokens/outcome, 44% of spend produced no booking) - framed as the scoreboard the other optimizations move, not a single fix. SCEN-008: supervisor under-uses find_places (90% no-delegation on place intent) and occasionally double-calls it (find_places,find_places) - tool-utilization grounding + de-dup, L3 prompt / L4 policy. Worked scenarios now span all dimensions + the maturity ladder: SCEN-001 (L3 prompt), 004/007 (L4/L5 policy), 003 (KPI), 008 (L3-L4 tool/routing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y effectiveness) SCEN-005: cost concentrates in a fat tail - the itinerary path costs ~8x a supervisor turn (22% of tokens from 4% of turns). Group-by on Debug.agent_path; feeds SCEN-003. L3 workflow reshape / L4 cost-aware routing. SCEN-002: memory-effectiveness gap - which memories improve outcomes. Honest caveat: recall volume is measurable (18% of turns) but per-memory outcome lift is NOT in current data; needs a small capture add (retrieved memory IDs per Debug turn) before it becomes a data-validated L4/L5 scenario. Catalog now has 7 worked scenarios spanning all 8 dimensions + the full maturity ladder; remaining candidate SCEN-006 (drift) stays parked (data flat). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d model selection First end-to-end optimization apply-loop (detect -> recommend -> apply -> verify), proven live against TravelAssistantV2. - optimization_policy.py: Cosmos-backed, versioned, reversible policy store (self-provisioning OptimizationPolicies container; proposed/active/reverted; short-TTL cache). Applying/reverting is a status flip + audit, never a code edit. - azure_open_ai.get_chat_model(): cached per-deployment model factory; reasoning models (gpt-5*/o-series) omit temperature and use api 2025-04-01-preview. - travel_agents.py: heuristic turn classifier (trivial/routine/complex) + per-tier prebuilt supervisor (shared tools+checkpointer). Default deployment when no active policy, so behavior is unchanged out of the box. - travel_agents_api.py: selects the tiered supervisor per turn; records model_tier/model_deployment on the Debug turn log. - optimization_recommendations.py + optimization_api.py: /optimizations REST surface (recommend card, propose/apply/revert). Prices are labeled ESTIMATES; the measured verify is authoritative. - optimization_mining.py --verify: per-tier token+cost report from Debug signal. Verified live: trivial->gpt-5-nano, routine->gpt-4.1-mini, complex->gpt-5.1; Debug records the actual serving model; revert returns to default. gpt-5-nano is a reasoning model (493 out-tokens on "hi") but input-token savings still make the trivial turn ~4x cheaper than mini. Deployed gpt-5-nano + gpt-5.1 to openai-kfpokdh52vbec (gpt-5.4 quota was fully allocated subscription-wide; later-gen models had free GlobalStandard quota). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ction) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Documents the core-app changes (02_completed), the learner target under the chosen pedagogy (pre-built plumbing + learner-built decision layer), updated/new learning objectives for an Analytics & Optimization module, the 01_exercises sync plan, and a self-contained maintainer change summary. Links it from the docs index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…imization) Resolve the 4 open decisions: new Module 07 (Lessons -> 08); supervisor-turn tiering core + worker sub-agent tiering as stretch; REST/CLI now + Angular card fast-follow; manual verify v1 + automated eval quality-gate as the autonomous capstone. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ons to 08 - New Module-07.md (Analytics & Optimization) teaching the apply-loop: detect -> build classify_turn_tier -> apply -> verify, plus stretch (worker tiering) and capstone (eval quality gate for autonomous apply). Carries a prerequisite banner for the v2 analytics baseline. - Renumber Lessons Learned 07 -> 08; update Home.md learning path and Module-06 nav. - workshop-integration.md: record verified finding that 01_exercises is behind 02_completed (no ADR-0007 Debug re-wire in the scaffold), making the v2 analytics baseline a prerequisite before the apply-loop code can be ported to 01. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Deliver the apply-loop into the exercise track WITHOUT touching Modules 01-05, via an additive layer that brings its own instrumentation. - infra (Bicep, provisioned by azd up): add OptimizationPolicies (/scenario) and OptimizationTurns ([/tenantId,/userId,/sessionId]) containers + gpt-5-nano and gpt-5.1 GlobalStandard deployments. az bicep build passes. - services/optimization.py: self-contained engine (policy store, model factory, select/get_supervisor_for_turn, record_optimization_turn -> OptimizationTurns, recommendations). classify_turn_tier ships as the learner STUB. No self- provisioning (Bicep owns containers); imports only existing 01 symbols. - optimization_api.py: /optimizations REST (recommend/propose/apply/revert). - Module-07.md rewritten for the additive flow: confirm Bicep tiers -> tour layer -> wire 4 hooks -> detect -> implement classifier -> apply -> verify -> stretch (worker tiering) -> capstone (eval quality gate). - optimization_mining.py --verify --container OptimizationTurns (flat schema; still supports --container Debug for 02_completed). - workshop-integration.md: record the additive-layer decision + implementation status; per workshop convention, all Bicep-deployed infra is pre-provisioned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scope for Module 07 (Analytics: instrument/detect/measure, L1-L2) and Module 08 (Optimization: apply/verify/autonomous, L3-L5), Lessons -> 09. Provided surfaces (Power BI + Optimization Console web app + Fabric/reverse-ETL); learners write loop logic, not UX; infra via Bicep. Includes per-module activity outlines, build backlog, coverage matrix, and open decisions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ra) - ALL PASS data/verify_live.py runs the REAL app code against cosmos-f2tx5x7js4bwi using DefaultAzureCredential (no keys), and cleans up after itself. Results: - B19: create_trip(session_id=...) persists sessionId (read back OK) - B1: store_node_executions provisions + writes NodeExecutions - engine reads node executions back from Cosmos - engine discovers the model-selection opportunity from LIVE data - cleaned up 121 verify docs Ledger B1 + B19 -> Grounded, verified live. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…L PASS
data/verify_analyst_live.py runs the app's Azure OpenAI (keyless/Entra) as the
analyst: for each detected opportunity it PROPOSES a card, which the engine
guardrails then validate/normalize. Live results:
- 2/2 cards accepted (bounded + cited); LLM chose correct seams
(config/model-selection for model-fit; prompt/supervisor.prompty for repeated-node)
- the LLM HALLUCINATED savings (\, \); the engine caught + overrode both
to the computed values (\.66, \) -> 'LLM proposes, engine disposes' proven
on real hallucinated output
Ledger B7 -> Grounded (safety + quality verified live). The biggest design risk
is retired.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ntic per-agent attribution Drive a real agent turn through /completion/stream and confirm against live Cosmos: node-grain telemetry is captured with correct semantic per-agent attribution (find_places/itinerary/supervisor) and the created Trip carries the driving session correlation key. Live verification surfaced that v2 sub-agents run nested under the supervisor's ReAct 'tools' node, so node-grain capture now prefers the sub_agent metadata tag over the raw langgraph_node for attribution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add engine/scorecard/ — a registry-extensible per-agent x dimension health rollup over node-grain telemetry. Scores only the dimensions node-grain can measure today (cost efficiency, model selection, workflow efficiency) and lists the other canonical dimensions with the signal each still needs, so nothing is fabricated. Rendered by data/agent_scorecard.py. Also align the engine's pure node_from_event with the app's semantic per-agent attribution (prefer sub_agent metadata over raw langgraph_node). Verified live against cosmos-f2tx5x7js4bwi: scored real NodeExecutions and flagged the supervisor premium-on-trivial model-selection opportunity. Self-test extended and green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…d superseded_recalled Add a self-test fixture proof that structural.repeated_node fires on an injected positive turn (back-to-back same sub-agent) and stays silent on a clean negative. Document the memory-side structural.superseded_recalled detector as deferred pending MemoryEvent-grain signal (per-recall memory identity + supersession state) that node-grain does not carry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…grain) Add engine/complexity/: realized_tier (measured output tokens) as a first-class primitive alongside a stdlib mirror of the app's keyword classifier, plus compare_coverage for a head-to-head. Self-test proves the measured signal has higher recall of truly-trivial turns (>=2 extra downgrade opportunities the conservative keyword tier misses) with zero false-downgrades. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add engine/seams/: a Registry of the app's optimizable seams (config domain, prompt files, code recipe). surface() returns the exact shape the analyst guardrails consume, and the reverse-ETL producer now sources it from here (removing the duplicated declared_surface). render_recipe() renders a concrete change instance: config binds params through the domain SDK to a fail-closed policy doc (auto/L4); prompt/code render staged, human-attested changes. Self-test extended. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rite) Add engine/codecontext/: a read-only retrieval interface with no write path, InMemoryProvider (testable) and FileBackedProvider (strict allowlist), plus scaffold_diff for a grounded, staged human-reviewed diff skeleton. Verified against the real repo: retrieved select_deployment_for_turn/classify_turn_tier from travel_agents.py, refused a non-allowlisted file, drafted an anchored diff. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…erified live Add engine/quality/: a reference-free LLM judge (model injected as an invoke callable so the engine stays import-clean), per-agent rubrics (supervisor / find_places / itinerary), the pluggable EvaluationResult primitive, and calibrate() reporting agreement/precision/recall vs human labels. Self-test calibrates a deterministic baseline within tolerance. Verified live (data/verify_quality_live.py): the real Azure OpenAI reference-free judge agreed with a per-agent labeled set at agreement 1.0 / precision 1.0 / recall 1.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rediscovery acceptance B5: detectors/statistical.cost_regression fires only on a statistically significant (z>=3), practically material (>=20%), and stable/consistent per-agent token regression, with a min-sample gate. Self-test proves: suppressed before N, silent on stationary baseline, not tripped by a single outlier, fires on a real regression; stays silent on the stationary simulator. B14: pipeline.rediscovered_scenarios maps discovered opportunities to catalogued SCEN ids; self-test acceptance shows the full pipeline rediscovers SCEN-007 end-to-end from telemetry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…1-C5) Add optimization_agent_api.py (separate small router, not bloating the legacy turn-centric file): agent scorecard, discovered opportunities (SLO-gated via a governance policy the engine consumes), staged-diff review (C2), approve/reject (C4), deploy-attest/confirm-revert (C1), SLO policy (C3), and learner schema declaration bound through the engine SDK (C5). Backed by a new self-provisioning OptimizationGovernance audit store and a node-exec query helper. Verified live against cosmos: scorecard, opportunities, and every C1-C5 action. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…C1-C5) Evolve the standalone console to the agents x dimensions model, split into small files (index.html shell + console.css + api.js client + console.js view). Renders the agent scorecard, engine-discovered opportunities (with SLO gate + governed state), and the C1-C5 governed actions: Review diff, Approve/Reject, Attest deploy/Confirm revert, the SLO policy form, and the declare-a-schema form. Verified end-to-end: api.js exercised live against the API, the view render functions produce correct DOM from live data (headless), and all files serve over http. Ledger C1-C5 marked Grounded (verified live). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add analytics/fabric/verify_engine_on_fabric.py: generates a verification
notebook, runs it as a job on the live F2 capacity, and reads results back
from Cosmos. Verified live on fabf2tx5x7js4bwi:
- B20: Fabric built-in gpt-5-mini runs on F2 via SynapseML OpenAIPrompt
(keyless, capacity-billed) and returns a real completion.
- B22: external Azure OpenAI (gpt-5.1) called KEYLESS via
notebookutils.credentials.getToken('https://cognitiveservices.azure.com')
-> 200 OK (retires the previously-open Entra-only auth risk).
- B21: a burst of 20 built-in calls completed 20/20 with no throttling
(SynapseML internal exponential backoff); F2 handles demo scale.
Infra finding: %pip fails in non-interactive RunNotebook jobs; use
pre-installed SynapseML + requests (no openai package).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…le fallback shipped The Power BI Option-A surface has two external gates that cannot be closed in this environment: building report visuals requires Power BI Desktop, and the translytical button write-path is a transient product bug (owner: fix ~mid-Aug 2026). The deployed UDF target exists. The agent-centric Console (now built and live-verified, C1-C5 Grounded) is the working fallback delivering the same loop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A polluted PORT (e.g. a stray PowerShell job object stringified into the env)
caused int(os.getenv('PORT', 8080)) to raise ValueError and abort MCP startup.
Parse defensively and fall back to 8080 with a warning.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ixes Replace the Optimization Console with a single-page web Analytics Portal (analytics/dashboard) served at /analytics/ from the frontend container, and make it the primary optimization surface across the solution and workshop. - Analytics Portal: 7 tabs (Overview, Optimizations, Model Selection, Memory, Agents, Business, Governance) reading the Travel API /optimizations/* live, with Live (recompute) and Reverse-ETL (notebook) sources plus Apply/Revert and governance actions. Baked into the frontend image via azure.yaml hooks. - Docs: rewrite USER_GUIDE and workshop Modules 06-10 to be portal-first and demote Power BI to an optional secondary report; add the Module 10 analytics recap and the Module 06 -> Module 10 exit ramp; add a root README analytics section. - Module 09: complete 7-tab portal rundown mapped to notebook sections, an edit-then-Run-all notebook flow, a self-contained LLM-analyst hands-on, and clarified that the mirror SQL endpoint is injected at provision time. - Optimization engine: LLM analyst (propose) + deterministic guardrails (dispose), tool-dedup projection, model-selection wiring; the Fabric reverse-ETL notebook now produces every dashboard metric. - Frontend: fix production build (typed catch in explore.component) and add 429 rate-limit handling. - Housekeeping: remove console/ folders; relocate workshop screenshots into 01_exercises/workshop/media; drop .DS_Store from tracking and gitignore OS junk. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
Mark Brown (markjbrown)
changed the title
…reset tooling - Remove the A/B demo seeder (ab_demo_seed.py) and retire the LLM data generators (data_generator.py, data_enricher.py, trivial_seed.py); inline the minimal Travel API client + Conversation model into marvel_seed.py so it stays self-contained. - Reframe demo-script.md to drive the model-selection before/after with the policy-aware traffic_simulator; trim the portal dataset dropdown to analytics/marvel and update the dashboard/analytics READMEs. - Remove demo_live_turns.py (superseded by traffic_simulator --mode app). - Consolidate/retire point-in-time docs (powerbi build-notes/report-delta, workshop lab-scope/integration, cosmos-throughput-analysis, verification/, optimization-scenarios/, MemoryIntelligence_Page_Spec) into the durable solution-architecture-guide; repoint the ADRs accordingly. - Move charter.md into docs/vision/. - Add reset_optimization_state.py and the portal "Freshen times" button/handler for demo data hygiene. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
- New analytics/powerbi/ (build guide, console theme, committed .pbix, and the legacy/ report templates moved under it); update the .gitignore pbix exception and provision_fabric.py's report-import default path. - New analytics/scripts/ (funnel_seed, marvel_seed, traffic_simulator, optimization_mining, Run-TrafficSimulator.ps1); fix each script's repo-root resolution (parents[1]->[2]) and the ps1's repo-root, and update the azure.yaml seed hooks plus every doc reference. - Remove the redundant rbac-mirror.ps1/.sh — provision_fabric.py already creates and assigns the custom FabricMirroringRole (readMetadata + readAnalytics) to the deploying user, workspace identity, and app identity automatically. - Rewrite analytics/README.md's file map; repoint USER_GUIDE, Modules 00/07/09, demo-script, the fabric READMEs, and ADRs 0008/0009 to the new paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
Add a hero shot of the Cosmos Voyager web app and an Analytics Portal Overview-tab screenshot to break up the text-heavy top-level README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
Mark Brown (markjbrown)
marked this pull request as ready for review
August 8, 2026 00:16
Mark Brown (markjbrown)
requested review from
Theo van Kraay (TheovanKraay) and
Aayush Kataria (aayush3011)
August 8, 2026 00:16
…rtal Demo tools Make the Analytics Portal fully functional without Fabric, and add discreet demo-maintenance controls that stay invisible in the workshop. - Relocate the OptimizationInsights builders from analytics/fabric/compute_insights.py into a new app service src/app/services/optimization_insights.py (single source of truth); slim compute_insights.py to a thin CLI wrapper that delegates to it. - Add POST /optimizations/insights — recompute the OptimizationInsights snapshot in-process (Fabric-independent), so the Business / Memory / Governance views light up with no notebook or mirror run. - Add POST /optimizations/reset — clear the runtime optimization state (governance + insights), reusing reset_optimization_state.py's logic via a demo_data service function. - Expose a `capabilities` flag on GET /optimizations/policies for client feature-detection. - Portal: replace the prominent Freshen button with a discreet gear (Demo tools) menu — Recompute insights, Freshen turn times, Reset optimization state. Recompute/Reset are feature-detected and hidden unless the API advertises them, so they never appear when the shared portal is served against the 01_exercises workshop API. Scope: 02_completed only (the workshop's optimization API lacks the recommendation engine). No deploy auto-run, preserving the Module-09 "run the notebook -> tabs light up" reveal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
…lds bake the current portal azd's ACR remote build excludes gitignored files from the upload context, so the frontend image kept baking a stale cached COPY layer (the portal was frozen before the Freshen button + Demo tools menu). Track the staged copy (kept in sync by the prepackage hook) so it is always in the build context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
…he live turns-by-minute chart moves build_turns_timeline and build_turn_metrics bucket Debug.timeStamp, but refresh_turn_times only re-stamped OptimizationTurns, so the live turns-by-minute chart stayed stale after Freshen. Re-stamp both containers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
…rator; model dedup Make the portal DRIVE the apply -> generate -> re-measure loop (not just observe it), so a reviewer can explore and reset with no CLI and no Fabric. - POST /optimizations/reset now also normalizes captured turns back to the single-premium baseline (in addition to clearing governance + insights), so the model donut shows one model and "apply model-selection -> tier" reads as a clean before/after. Tokens, the funnel signal, and app data are untouched; only non-baseline turns are rewritten. - POST /optimizations/traffic generates policy-aware synthetic turns (baseline single-model until model-selection is applied, capability-tiered once active), dual-writing Debug + OptimizationTurns so every live view reflects it — the in-process equivalent of traffic_simulator.py --mode direct. - Fix the model-distribution duplicate: canonicalize versioned model names (gpt-5.1-2025-11-13 -> gpt-5.1) so the same model isn't split across two donut slices. - Portal: the discreet gear "Demo tools" menu gains Generate traffic; Reset becomes Reset-to-baseline; both are feature-detected via the /policies capabilities flag so they stay hidden against the 01_exercises workshop API. - Docs: add a "Demo tools (gear)" section + reviewer quick loop to USER_GUIDE.md (with zero-setup notes on Acts 4/5); refresh the demo-script.md gear-menu note. 02_completed only — the workshop's simpler optimization service has no recommendation engine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 40814ead-403b-4d65-8866-9424d08d91bb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Two intertwined efforts, one unifying PR to
main— grounded inanalytics/docs/vision/agent-analytics-and-optimization-vision.mdandanalytics/docs/vision/charter.md, and superseding the first attempt (#73):agent_memory_toolkit_v2— replace the classic orchestrator + specialist-nodes architecture with the v2 model: supervisor + sub-agents-as-tools, theazure-cosmos-agent-memorySDK, and thelangchain-azure-cosmosdbcheckpointer.maincurrently reverted the memory-toolkit work, so this PR carries the full classic → v2 transformation. (ADR-0006.)The Agent Analytics & Optimization plane
Built across all six pillars from the vision — Agent Performance, Agent Collaboration, Cost Intelligence, Memory Intelligence (flagship), Evaluation Intelligence, Workflow Intelligence — at Levels 1–3 across the pillars, one bounded, audited-and-reversible Level 4 slice (memory-retention / model-selection), with Level 5 as conceptual framing (per the charter).
analytics/dashboard, served at/analytics/) — the operational optimization surface: pillar views reading the Travel API/optimizations/*live, with Live (recompute) and Reverse-ETL (notebook) sources, one-click Apply / Revert, and a governance lifecycle.OptimizationInsightsrows back to Cosmos; an LLM analyst proposes recommendation cards and deterministic guardrails dispose (the engine computes the saving); optional translytical Apply/Revert via a Fabric User Data Function.v2 adoption details (ADR-0006, carried from #73)
find_places,create_or_update_itinerary) replace the orchestrator + five specialist nodes.azure-cosmos-agent-memorySDK owns extraction / dedup / summaries / supersession (replaces the hand-rolled memory subsystem).langchain-azure-cosmosdbcheckpointer.cosmosdb-gsi.bicep,seed_gsi_trips.py) is carried forward from v2 but gated behindDEPLOY_GSIand not used in the default deployment.Bug fixes (supersedes #69–#72)
Delivered on v2:
recall_memories(a tool list was invoked as one tool, so memories were never read); saved trips landing under a hallucinateduser_id(fixed via identity injection); Windows UTF-8/emoji logging crash; real token + agent/hand-off capture via the re-wired CosmosDebugpath; frontend production-build fix (typedcatch) + 429 rate-limit handling.Related
search_cosmosstatus/metadata predicate so pruned memories are excluded inside the Cosmos vector query (backs the Module-08 memory-retention optimization). PR to follow once the API shape is confirmed.Notes for reviewers
02_completed/; the build-along workshop in01_exercises/workshop/; the knowledge base (vision, charter, ADRs, deep docs) inanalytics/docs/.Why this PR is large (377 files) — and how to review it efficiently
This is a first-time unification of three long-lived lines of work into
main, which is still ata70a8e1(PR #67). None of the prerequisites were ever merged tomain— they landed on integration branches, so they all show up here:agent_memory_toolkit_v2(PRs Perf improvements: supervisor + @tool sub-agents #57, update readme #59, Updating Module 01-06 #64, Updating checkpoint to use the new azure-langchain package #65 — already reviewed) → ~95 files.analytics(PRs Add Fabric Analytics #48, update #49, Performance Improvements for Travel Multi-Agent Workshop #51, Sync analytics branch with latest main + fixes #68) → ~40 files.The count is also inflated ~2× by the mirrored trees:
01_exercises/(130 files) and02_completed/(136 files) carry largely the same edits — the teaching scaffold vs. the completed solution.Efficient review path — to see the net-new work without the mirror duplication or the already-reviewed toolkit commits:
analytics/(107 files) — the new Analytics & Optimization plane (portal, Fabric reverse-ETL, engine, docs).02_completed/— the completed solution; you can skip the01_exercises/mirror, which tracks the same changes as the teaching scaffold.