Feature/stable drupal #861
Open
+12,751
−5,742
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…to feature/drupal-migration
…tack/migration-v2 into feature/drupal-migration
… in Drupal services - Added `mysql2` package to dependencies for improved database interactions. - Refactored asset configuration handling in various services to use a unified `assetsConfig` structure instead of `drupalAssetsUrl`. - Enhanced validation processes to include asset accessibility checks. - Updated related service methods to accommodate new asset configuration structure. - Improved logging for better traceability during asset validation and database interactions.
…ling - Introduced new AEM migration services, including content type and locale processing. - Updated asset configuration to utilize environment variables for dynamic base URL and public path. - Refactored file processing and validation logic to accommodate new AEM structure. - Enhanced logging and error handling for improved traceability during migration processes. - Removed deprecated package-lock files for other migration services.
- Updated import statements to use single quotes for consistency. - Refactored functions in `projects.contentMapper.controller.ts` for improved readability. - Added `getExistingTaxonomies` function to retrieve taxonomies from both source and destination. - Enhanced `putTestData` in `contentMapper.service.ts` to store taxonomies in the project database. - Updated UI components to support taxonomy selection and display. - Improved logging for better traceability during taxonomy operations.
… component - Introduced loading state for taxonomies to provide user feedback during data fetching. - Conditional fetching of taxonomies based on field type to optimize performance. - Removed unnecessary debug logging to clean up the code and improve readability. - Updated UI elements to reflect loading status and handle empty taxonomy states more gracefully.
…nce and readability - Added useCallback to handleSelectedEntries in ContentMapper for better performance. - Updated pluralization logic in ContentMapper for cleaner code. - Enhanced log parsing in LogScreen components to skip empty lines and silently handle malformed entries, improving robustness.
…vice and UI - Removed unnecessary console logs from contentMapper.service.ts to enhance performance and readability. - Updated field type handling in content-types.service.ts to prioritize user-selected types over defaults. - Added logging for field type changes in the ContentMapper component to track modifications sent to the backend. - Cleaned up code by eliminating redundant logging statements across various functions.
- Introduced a priority system for field type mapping, first checking the generated content type schema for user-selected field types. - Improved error handling when loading content type schemas, ensuring fallback mechanisms are in place. - Cleaned up redundant code related to field type inference, streamlining the process for better performance and readability.
- Added optional assetsConfig field to LegacyCMS interface for better asset management. - Updated putTestData function to conditionally update assetsConfig only if provided, preventing overwrites with empty values. - Refactored updateFileFormat to ensure existing asset configurations are preserved when updating. - Removed unnecessary console logs in createDrupalMapper for cleaner code and improved performance.
…rvice - Added logging to putTestData for initializing reference and taxonomy fields with their respective referenceTo values. - Implemented logging in updateContentType to track updates for reference and taxonomy fields, improving traceability during field mapping operations.
…to feature/drupal-migration
…lated logic - Updated various services and components to exclude 'profile' fields from processing and mapping. - Enhanced the content type mapper to filter out profile references and prevent their inclusion in generated files. - Fixed typos in reference handling across components for consistency. - Added new test scripts for profile reference validation in both API and upload-api packages.
- Added 'project' parameter to various service functions to support dynamic locale mapping. - Implemented a new utility function, mapDrupalLocales, to facilitate mapping of source locales to destination locales based on user-selected configurations. - Updated processEntries and createLocale functions to utilize the new mapping logic, improving flexibility in locale transformations. - Enhanced overall code readability and maintainability by restructuring locale handling processes.
- Added AssetUrlTracker interface to monitor successful and failed asset downloads. - Enhanced saveAsset function to utilize URL tracking, logging both successful and failed attempts. - Implemented fallback path logic for asset downloads, improving reliability in case of primary path failures. - Updated createAssets function to initialize and write asset URL summary to assets_url.json, providing better visibility into asset processing outcomes.
- Changed the title of the SQL file format in legacyCms.json to "ApiTokens". - Refactored LoadFileFormat component to enhance SQL connection detection using multiple indicators. - Improved file format validation logic for Drupal SQL connections, ensuring correct handling and icon display. - Updated Redux state management for selected file formats to reflect changes in SQL connection handling.
- Removed specific Rice University path from asset detection logic for better generalization. - Enhanced validation message handling in LoadUploadFile component to provide specific error messages for SQL connections. - Updated database configuration to use 'riceuniversity1' instead of 'riceuniversity2'. - Refactored file processing logic to handle both old and new validation result formats, providing clearer error messages for database connection issues. - Improved drupalValidator to return detailed error messages for various connection failures, enhancing debugging capabilities.
…ment and logging - Added detailed console logging in FileComponent and LoadUploadFile for better debugging of file details. - Ensured fileDetails are always updated from Redux to prevent stale data. - Fixed data spreading in newMigrationData to prevent data loss during updates. - Updated validation message handling to utilize Redux state for isSQL checks, improving accuracy in connection status reporting.
- Added new properties to the Project interface for better locale management, including source_locales, master_locale, locales, and localeMapping. - Implemented logic to reconstruct localeMapping from master_locale and locales during migration updates. - Enhanced the updateLocaleMapper function to save locale mappings effectively, including detailed logging for successful updates. - Improved the processEntries and mapDrupalLocales functions to utilize new locale mapping logic, ensuring accurate locale transformations. - Updated UI components to save locale mappings to the backend immediately after user selections, enhancing user experience and data consistency.
- Enhanced ProgressBar component by adding default values for props, improving usability. - Updated type definitions for better type safety in calculations related to circular progress. - Simplified the component structure by removing redundant defaultProps assignment. - Adjusted BrowserRouter configuration to enable new transition features for improved routing performance. - Minor fix in MainHeader component to streamline Tooltip usage.
…configuration for TypeScript, and enhance error handling in various services
…le is consistently set as the first element, and improve logging for better debugging
…to feature/drupal-migration
…in LoadLanguageMapper - Added validation for field type conversion in `convertToSchemaFormate` to prevent invalid type changes. - Removed unnecessary console logs and added more informative logging in `LoadLanguageMapper` for better debugging. - Updated configuration to switch CMS type to 'drupal', set local path to false, and enable SQL support.
…geMapper - Added detailed logging to track state changes and auto-mapping processes in `LoadLanguageMapper`. - Updated locale mapping logic to ensure immediate application of parent locale state and preserve auto-selected locales. - Changed CMS type to 'contentful' and set local path to true in configuration for better development setup.
… and debugging - Introduced a new useEffect to update existingLocale when cmsLocaleOptions changes, ensuring accurate initial population. - Enhanced logging for better tracking of locale synchronization and state changes. - Updated dependencies in useEffect to prevent infinite loops and ensure proper processing of locale mappings. - Improved handling of master locales and added checks for existing mappings to streamline the auto-mapping process.
…ources in LoadLanguageMapper - Added checks for exact destination matches to improve locale mapping accuracy. - Implemented handling for unmatched source locales, ensuring they are assigned to available destinations. - Enhanced logging for better visibility into the mapping process and state changes. - Updated cmsLocaleOptions to guarantee the master locale is always listed first.
- Updated .gitignore to include new file types and directories. - Removed package-lock.json from the repository. - Added MySQL2 dependency to package.json for database interactions. - Introduced new ESLint configuration for the API. - Created nodemon.json for development environment settings. - Refactored various service files to improve error handling and logging. - Implemented new Drupal services for content extraction and mapping. - Enhanced existing services with better type handling and validation. - Updated UI components for improved user experience and functionality. - Added batch processing utilities for efficient data handling. - Improved overall code structure and documentation for better maintainability.
- Updated UID processing logic in AEM and Sitecore services to ensure correct field mapping. - Enhanced error handling in AdvanceProperties component to provide user feedback for validation regex input. - Cleaned up unnecessary comments and improved code readability in ContentMapper and LegacyCms components. - Added default value for affix in file validation to streamline user experience. - Updated legacy CMS description for clarity and improved user guidance.
- Added flexbox styling to .table-row for better alignment and spacing in ContentMapper component. - Updated LoadFileFormat component to improve file format icon handling and accessibility. - Refined file processing logic to restrict XML handling to specific CMS types (WordPress and Drupal).
- Implemented sanitization of stackId to prevent path traversal vulnerabilities. - Added validation to ensure resolved file paths remain within the allowed directory. - Updated file reading logic to use the sanitized and validated path.
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request implements comprehensive Drupal SQL-based migration support, enabling direct database connectivity as an alternative to file-based migrations. The feature adds SQL validation, connection management, and dynamic query generation specifically for Drupal 8/9/10/11 databases.
Key Changes
- Drupal SQL Validator: New validator for MySQL connections with asset URL validation
- SQL Connection Support: Added MySQL connection configuration throughout the application stack (upload-api, API, UI)
- Dynamic Query Generation: Optimized query builder to handle complex Drupal schemas without JOIN limits
- UI Enhancements: Updated file format detection, validation flows, and locale mapping for SQL sources
- Locale Normalization: Consistent lowercase locale handling across all CMS types
Reviewed changes
Copilot reviewed 80 out of 85 changed files in this pull request and generated 36 comments.
Show a summary per file
| File | Description |
|---|---|
| upload-api/src/validators/drupal/index.ts | New Drupal SQL validator with connection testing and asset validation |
| upload-api/src/services/fileProcessing.ts | Added SQL file processing support with database validation |
| upload-api/src/services/drupal/index.ts | Drupal mapper service for SQL-based migrations |
| upload-api/src/helper/index.ts | Added MySQL connection helper functions |
| ui/src/components/LegacyCms/Actions/LoadUploadFile.tsx | Updated file upload to handle SQL connections |
| ui/src/components/LegacyCms/Actions/LoadFileFormat.tsx | Enhanced file format detection for SQL and directories |
| ui/src/pages/Migration/index.tsx | Updated migration flow with locale auto-mapping |
| api/src/services/drupal.service.ts | Modular Drupal migration service with SQL extraction |
| api/src/utils/optimized-query-builder.utils.ts | Query optimization to handle large Drupal schemas |
| api/src/services/projects.service.ts | Added MySQL details storage and query generation |
Files not reviewed (2)
- api/package-lock.json: Language not supported
- upload-api/package-lock.json: Language not supported
Comments suppressed due to low confidence (1)
api/src/utils/entries-field-creator.utils.ts:446
- This case label is a duplicate of 'isodate'.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
…to feature/stable-drupal
- Updated migration service to use the correct destination stack ID. - Enhanced AdvancePropertise component to include additional toggle states in field settings. - Modified ContentMapper styles for better layout, including a sticky footer for the save button and adjustments to prevent header wrapping. - Improved file processing error handling in upload API to ensure robust responses and logging.
…nd improving step completion logic - Added logic to set a default affix value of 'cs' if not already defined in the LoadPrefix component. - Updated the LoadSelectCms component to preserve or set the default affix when selecting a CMS. - Modified the Migration page to ensure the affix is correctly assigned from project data or defaults to 'cs'. - Improved step completion checks in the LegacyCMSComponent to include affix validation.
…rts in locales service - Simplified the validation logic for searchText in the getAuditData function of the migration service by removing redundant null checks. - Cleaned up the import statements in the locales service by removing the unused LOCALE_MAPPER import.
- Updated customLogger calls in projects.service.ts to handle stackId more consistently. - Simplified value handling in entries.service.ts to avoid unnecessary null checks. - Adjusted contentTypeMapper.js to initialize displayType more clearly. - Modified drupal validator to streamline database query execution and logging.
…ionality - Removed redundant audit log index calculations in migration.service.ts. - Cleaned up import statements in entries.service.ts by removing unused constants. - Enhanced uidCorrector function to support a default prefix for UID generation. - Streamlined error handling and logging in various Drupal services for better maintainability.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.