Conversation
The Copilot Agent Interactions.pbit file was updated. Specific changes within the Power BI template are not detailed, but the binary content has been modified.
Updated "get or create" helper methods to first search for existing entities in the local context before querying the database. This prevents duplicate entries and unnecessary DB queries by reusing unsaved entities. Also clarified comments and removed redundant SaveChanges calls from these methods.
- Convert Add, AddRange, and Flush to async Task methods using SemaphoreSlim for async locking - Make batch processing logic fully async and thread-safe - Update all usages and tests to await async methods - Simplify ParallelListProcessor with async/await and remove custom thread unlock logic - Ensure thread-safe buffer management and correct parallel execution in tests - Improves reliability and modernizes batch processing utilities for async .NET patterns
Introduced the Tests.StressTesting project for memory and performance stress testing of the Office365ActivityImporter.Engine pipeline. Includes interactive console menu, configurable test parameters, memory monitoring, and reporting. Added fake loader implementations and infrastructure for extensible stress tests. Updated solution file and made minor changes to UnitTests for public accessibility.
Refactor AbstractUserAppLoader to await AddRange and Flush methods of ListBatchProcessor, improving scalability and efficiency by leveraging asynchronous processing for large user lists.
Added <gcAllowVeryLargeObjects enabled="true"/> to App.Template.config to allow allocation of arrays and objects larger than 2GB. This change supports scenarios requiring processing of large datasets or files.
Removed redundant CreateOrUpdateRunbookConfigureTask and its project reference. Added profiling PowerShell scripts and documentation to WebJob.Office365ActivityImporter project so it's seen in VS solution tree, including details on SQL script handling and profiling logic in readme.md.
Refactored parallel processing to materialize enumerables and use efficient chunking, reducing repeated enumeration and memory spikes. Lowered chunk and batch sizes in import pipeline to prevent OutOfMemoryException with large datasets. Switched report loading to stream-based reading and stopped storing full JSON in each log item. Explicitly dispose HTTP responses to free memory promptly. Added comments explaining memory management changes.
Added null check and default "Unknown" value for AppHost in CopilotAuditLogContent to prevent null reference exceptions when CopilotEventData is missing.
Refactored activity log import to reduce memory usage and improve error handling: - Stream and parse JSON directly from HTTP response streams to avoid OOM errors. - Increase parallel chunk size for report loading from 250 to 1000. - Refactor deserialization to use JToken, minimizing string conversions. - Improve error handling: log and skip bad reports instead of throwing. - Replace recursive metadata pagination with an iterative loop. - Simplify date formatting and debug trace file name sanitization. These changes make the importer more reliable and scalable for large datasets.
Refactored UserMetadataUpdater to improve maintainability and memory management by extracting batch processing, license/SKU assignment, and user data mapping logic into dedicated helper classes: UserBatchProcessor, UserLicenseProcessor, and UserDataMapper. Updated main user import flow to use these helpers and process users in batches. Added REFACTORING_SUMMARY.md documenting rationale and benefits. Updated project file to include new classes. All public APIs and tests remain backward compatible.
Refactored all lookup cache classes to inherit from a new base class (DBLookupCacheForEntityWithName<T>) that centralizes entity loading logic and uses FirstOrDefaultAsync with OrderBy to gracefully handle existing duplicates. Enhanced GetOrCreateNewResource with robust DbUpdateException handling to prevent duplicate inserts and FK constraint errors during batch processing. Introduced DetachAllEntitiesExceptLookups to maintain lookup cache consistency across batches, and updated batch processors to use this method. Added comprehensive unit and integration tests (DBLookupCacheTests, LookupCacheBatchProcessingTests) and documentation to ensure reliability, cache consistency, and future maintainability. Removed redundant Load implementations from individual caches.
Added DBLookupCacheDuplicateKeyErrorTests.cs with 9 targeted unit tests to validate the FK constraint violation fix for large-scale user imports. Registered the new test file in the project and added detailed documentation (Complete_TestSuite_Summary.md, DBLookupCacheDuplicateKeyErrorTests.md, EntityFramework_LINQ_Query_Fix.md) covering test coverage, troubleshooting, and EF best practices. Refactored all test code to evaluate string interpolations before LINQ queries, ensuring Entity Framework compatibility and preventing translation errors. Updated LookupCacheBatchProcessingTests.cs cleanup logic accordingly. These changes ensure robust validation of the fix, improved test reliability, and comprehensive documentation for future maintenance.
Resolve duplicate key errors when importing users and managers from external sources by ensuring all navigation property assignments use tracked EF entities. - Add comprehensive unit tests for cross-batch, large batch, and AAD ID mismatch scenarios. - Introduce helpers to always use tracked user entities in UserBatchProcessor and UserDataMapper. - Fallback to UPN-based lookup if AAD ID lookup fails, preventing duplicate user creation. - Improve comments and documentation for maintainability. This fixes a critical production bug and ensures robust, batch-safe user import logic.
Updated BatchProcessing_WithRepeatedLookups_NoFKViolations to generate a single testRunId at the start of the test. This testRunId is now used for all generated department names, location names, and user principal names, ensuring consistency across batches. Also updated the user search pattern in assertions to use testRunId, preventing mismatches caused by multiple DateTime.Now.Ticks calls. This improves test reliability and repeatability.
Marked WebApiStatsUploader as obsolete and removed all usage stats upload logic from Program.cs, including database, Redis, and WebApiStatsUploader integration. This reflects the deprecation of the stats uploading feature.
- Split UserMetadataUpdater tests into focused files: Basic, Insert, License, Manager, and Metadata, improving organization and maintainability. - Added UserMetadataUpdaterMetadataTests covering metadata enrichment, nullification, whitespace trimming, and lookup entity reuse. - Improved batching performance in user/manager/license processing by replacing .Skip().Take() with .GetRange(). - Optimized manager resolution with O(1) AAD ID lookups and ensured tracked entities are used to prevent duplicate key errors. - Enhanced handling of null/cleared metadata fields and duplicate UPNs. - Updated .csproj to include new test files and removed old test region from UserImportTests.cs.
…eRange with bulk SQL DELETE The license removal step during user metadata import was taking 10+ hours for ~187K users due to two bottlenecks: 1. Loading license lookups individually per user via db.Entry(user).Collection(...).Load() — 187K separate SQL queries 2. EF RemoveRange generating individual DELETE statements per entity Replace both with a single bulk SQL DELETE and remove the per-user load since tracked entities are no longer needed for removal.
Implement high-performance bulk update for existing users using temp table and SQL UPDATE JOIN, bypassing EF per-entity tracking when tenant SKUs are available. Add UserMetadataUpdaterPerformanceTests to measure insert/update throughput and verify correctness. Optimize license lookup creation to use UserId FK directly. Expose GraphUsersByAadId for efficient lookups. Update tests and loader to support new logic. These changes greatly improve scalability for large tenants.
Move two-phase user insert (bulk + enrich) from UserMetadataUpdater to new UserInsertProcessor class for better separation of concerns. Update UserMetadataUpdater to delegate to the new class. Add UserInsertProcessor.cs to project. Improves maintainability, testability, and code organization.
Enhance logging to clearly distinguish between new and existing users during import and metadata enrichment. Add null checks before clearing LicenseLookups to prevent exceptions. Only include LicenseLookups when tenant SKUs are unavailable to reduce memory usage and avoid OOM errors. Refactor existing user update logic for better error handling and reporting. Improve final summary logs to show counts of inserted and updated users.
Added logic to enable SCM and FTP publishing credentials for Azure App Service websites. This includes creating and updating publishing credentials policies to allow basic authentication, with log messages indicating the actions performed.
Added validation to ensure CognitiveEndpoint and CognitiveKey are set in AppConfig before running CommentsCognitiveTests. The test is now marked inconclusive if required configuration is missing, preventing false failures.
Move cognitive config validity check outside the per-date loop when loading message cognitive stats for a channel. This avoids redundant checks, logs a warning and adds basic stats if the config is invalid, and improves code clarity and efficiency.
Updated AppInsightsAuthTests to require ApplicationId in connection strings and adjusted test logic accordingly. Improved GraphImportTests to handle missing cognitive config by marking sentiment assertions inconclusive instead of failing.
ChatMessageLoadCognitiveStatsFromCacheOrAITests now checks if the cognitive configuration is valid before running. If not, the test is marked inconclusive to avoid failures due to missing config values.
Replaced all SPFx Log usage in AiTrackerModernApplicationCustomizer.ts with a new Logger class that logs to both the browser console and SPFx Log, ensuring visibility in production. Added Logger.ts with static logging methods. Updated imports and removed redundant LOG_SOURCE constant from the main class. Incremented solution version to 1.0.1.58 and updated the .sppkg package.
• Add RoleAssignmentTask to assign RBAC roles to Azure resources in a resource group • Extract ServicePrincipalResolver to resolve object ID from client credentials, shared by RoleAssignmentTask and KeyVaultTasks • Add ResourceSecurityInstallJob to assign Reader role to the runtime account • Integrate ResourceSecurityInstallJob into SolutionInstaller, continuing on failure • Add Azure.ResourceManager.Authorization package dependency
jesusfer
approved these changes
Mar 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fix primarily addresses this deprecation: https://azure.microsoft.com/en-us/updates?id=transition-to-azure-ad-to-query-data-from-azure-monitor-application-insights-by-31-march-2026#:~:text=Published%20date%3A%20May%2015%2C%202023,application%20insights%20will%20be%20retired
In addition, scripts loaded in the browser are now CSP compliant: https://learn.microsoft.com/en-us/sharepoint/dev/spfx/content-securty-policy-trusted-script-sources
Extras:
Installer
App service created with publishing profiles enabled to FTP works.
Installer adds "read" role for runtime account to resource group. Needed for Application Insights access.
WinForms Project Clean-up
NewSaltFormand its associated designer and resource files from the WinForms project, cleaning up the codebase and project configuration. [1] [2] [3] [4]Documentation Update
usp_CompileWeekly.