feat: [Team] ONCEHUB-98788: updating transformer version to 1.102.2#98
feat: [Team] ONCEHUB-98788: updating transformer version to 1.102.2#98sumitsheokand merged 308 commits intoteam/technocratsfrom
Conversation
…(#4427) * docs: add usage documentation for environment variable override feature - Add comprehensive documentation for envOverrides usage in component tests - Include examples for API endpoint override, debug mode, and feature flags - Provide best practices for dynamic vs static environment variable usage - Document common environment variables and implementation details - Guide developers on proper usage to ensure test compatibility * feat: integrate environment variable override into component test framework - Move envUtils.ts to test/integrations/ directory alongside testUtils.ts - Integrate EnvManager into component.test.ts execution flow - Add environment variable handling with proper cleanup in finally blocks - Update rudder_test destination to support dynamic RUDDER_TEST_API_ENDPOINT override - Use runtime function evaluation instead of module load time for proper test isolation - Add working example test case in rudder_test processor data.ts - Update main README with comprehensive environment variable override documentation - Follow component test patterns by using data.ts files only Environment variable overrides now work seamlessly in component tests with: - Automatic snapshot and restore of environment variables - Per-test isolation with proper cleanup - Real working example demonstrates API endpoint override functionality - Dynamic evaluation ensures environment changes take effect during test execution * chore: remove unnecessary envUtils.README.md file The documentation has been integrated into the main README.md file, so the separate documentation file is no longer needed.
* fix: enhance config object in ga4v2 instead of mutation * chore: update changelog
…n (#4322) * feat: onboarding facebook value based audience * chore: minor change * fix: minor issue * fix: addressing comments * fix: addressing comment * chore: addressing comments * fix: sonar issue
* fix: handle empty oauth secret for multiple destination (#4295)
* fix: handle empty oauth secret for multiple destination
* fix: test
* fix: add validation for other field
* chore: add missing test
* chore: add more test
* chore: change test
* chore: misc
* feat: implement hasDynamicConfig flag support (#4321)
* feat: implement hasDynamicConfig flag support in routerTransform flow
This implements support for the hasDynamicConfig flag in the routerTransform flow to fix
the issue where events with dynamic configuration were being mixed up during transformation.
The implementation includes:
1. Added a new hasDynamicConfig flag to the Destination type in controlPlaneConfig.ts to indicate
whether a destination's configuration contains dynamic config patterns
2. Modified the dynamicConfigParser.ts to respect the hasDynamicConfig flag:
- Skip processing when hasDynamicConfig is explicitly false
- Process events when hasDynamicConfig is true or undefined (for backward compatibility)
3. Enhanced groupRouterTransformEvents function in src/v0/util/index.js to:
- Group events by destination config when hasDynamicConfig is true or undefined
- Use fast-json-stable-stringify to ensure consistent grouping of configs with different key orders
- Maintain backward compatibility with older server versions where the flag might not be present
4. Added comprehensive test cases for:
- Testing behavior with hasDynamicConfig=false
- Testing behavior with hasDynamicConfig=true
- Testing backward compatibility with hasDynamicConfig=undefined
- Testing mixed hasDynamicConfig flags in a batch
- Testing configs with different key orders
This implementation resolves INT-3633 by ensuring that events with dynamic configuration are properly
grouped and processed in the routerTransform flow, preventing event mix-up issues.
Dependencies:
- Added fast-json-stable-stringify package to ensure consistent JSON string representation regardless of key order
* fix: add hasDynamicConfig flag support in routerTransform flow
* test: add hasDynamicConfig flag to test files
fix tests
* refactor: optimize dynamicConfigParser to avoid deep cloning entire event
* fix: restore unset functionality in dynamic config parser
- Add unset-value import to properly remove extracted fields from events
- Restore unset() call after extracting dynamic config values to prevent downstream issues
- Change from shallow to deep cloning to avoid modifying original events during unset operations
- Add comprehensive test to verify unset functionality works correctly
- Add shouldGroupByDestinationConfig method to JavaScript dynamic config module
- Update groupRouterTransformEvents to use shouldGroupByDestinationConfig function
- Add environment variable tests for USE_HAS_DYNAMIC_CONFIG_FLAG in router grouping logic
This ensures that fields used in dynamic config templates (e.g., {{ message.traits.appId || 'default' }})
are properly removed from events after extraction, preventing potential conflicts in destination processing.
* refactor: optimize dynamic config parser and consolidate utilities
- Move utility functions from dynamicConfig.js to utils.js for better organization
- Optimize DynamicConfigParser with pre-compiled regex patterns and efficient loops
- Replace deep cloning with selective cloning for better performance
- Remove standalone dynamicConfig.js and dynamicConfig.test.js files
- Update imports and references across affected modules
* fix: ensure consistent ordering in batch processing for google adwords offline conversions (#4359)
- Add sorting by jobId to removeDuplicateMetadata function
- Sort final batch results by minimum jobId for consistent ordering
- Re-enable router output assertion for google_adwords_offline_conversions tests
- Update test data ordering to match new consistent batch processing behavior
- Fix unit test expectations to match new sorted metadata order
* chore: add concurrent requests and memory fencing middlewares (#4358)
* chore: migrate tiktok_ads to new test (#4342)
* chore: migrate tiktok_ads to new test
* chore: review comments
* chore: review comments
* chore: review comments
* chore: review comments
* chore: rebase
* chore: introduce deep freeze to compaction middlewares (#4365)
* feat: implement rudder-test destination with migrated tests (#4352)
- Add rudder-test destination for platform testing and development
- Implement TypeScript destination with router-only transformation
- Support Record v2 events with echo response pattern
- Add context-based test behavior control via testBehavior field
- Migrate tests to new format using migration script
- Optimize tests with generateMetadata utility and common destination
- Enable Zod validation and enhanced Allure reporting
- Add npm script 'rudder-tests' for convenient test execution
- Update README.md with comprehensive destination documentation
The rudder-test destination provides a controlled testing environment
without external dependencies, ideal for platform feature testing,
development workflows, and CI/CD reliability.
refactor: reduce code duplication in rudder_test destination
- Extract common validation and request building logic to utils.ts
- Add processor transformation function alongside existing router function
- Create comprehensive component tests for both processor and router
- Implement shared utilities for message validation, request config building, and test behavior handling
- Maintain different error handling patterns for processor (throws) vs router (returns error objects)
- Add 6 test cases covering success scenarios, error responses, and validation failures
refactor: improve type safety in rudder_test utils
- Replace 'any' types with specific Metadata and RudderTestDestination types
- Import proper types from '../../../types' for better type safety
- Enhance IntelliSense support and compile-time error detection
- Address coderabbitai bot feedback on PR #4352
feat: enhance rudder_test destination with comprehensive dynamic config testing
- Modified rudder_test destination to actually use resolved dynamic config values
- Added endpoint and apiKey resolution from destination config in utils.ts
- Updated transform.ts to pass destination config to utility functions
- Enhanced type definitions to support RudderTestConfig interface
- Added multi-event test cases for both processor and router
- Tests validate multiple events with different dynamic config values resolve correctly
- Added mixed destination ID scenarios (different hasDynamicConfig flags)
- Added compaction tests with multiple events and dynamic config
- All 20 tests now pass and actually validate dynamic config resolution
These tests will surface thread safety issues and shared state problems
in dynamic config processing by testing scenarios where multiple events
have different template variable values that must resolve independently.
docs: update rudder_test destination documentation with dynamic config features
- Added dynamic config resolution capabilities to feature list
- Updated implementation details to reflect both processor and router support
- Added comprehensive test coverage information (20 test cases)
- Included dynamic config examples with template syntax
- Added multi-event testing and thread safety testing use cases
- Updated key files list to include utils.ts and enhanced type definitions
- Corrected architecture description from router-only to dual implementation
* chore: sfmc ai generated docs (#4328)
* chore: sfmc ai generated docs
* chore: sfmc api section in docs
* chore: separate out transformer logic from main doc
* chore: vdm-v2 related faulty docs fix
* Revert "Revert "Merge branch 'develop' into main""
This reverts commit af6e580.
* fix: klaviyo error logging
* chore: optimize usage of deepFreeze in compaction middlewares (#4374)
* chore: use batch processing in transformations (#4363)
* feat: use account information on zoho (#4361)
* chore: new way of creating release
* fix: deep freeze in compaction middleware
- Add test behaviors in rudder_test transform to mutate and replace destination config
- Add router and processor component tests for both property mutation and config
reference replacement in compacted payload flow
- Assert correct error handling and status codes when mutation is attempted on frozen config
- Ensures immutability of config is enforced and covered in both flows
- Remove unneccessary clonedeep in router transform flow by fixing code
* chore: add safe batch processing to avoid blocking in highly batched destinations
* chore: add batching in mp and source transformations
* fix(marketo): remove auth key from logs
* feat: testing tool (#4391)
* feat: testing tool
* chore: changes
* chore: code structure
* fix: eslint
* chore: review comments
* chore: review comment
* feat: add server request creation flow
* chore: bluecore add internal doc and migrate test (#4353)
* fix: blucore optin event mapping
* fix: review comments
* chore: add subscription event
* chore: add subscription event
* fix: reviews comments
* chore: blucore add internal doc and migrate test
* chore: misc
* chore: misc
* fix: readme
* chore: update docs
* fix(bluecore): address PR #4353 reviewer feedback and API documentation corrections
✅ Complete reviewer feedback resolution:
- Removed all 16 'NEEDS REVIEW' placeholders with concrete information
- Removed AI-generated optimization recommendations section
- Simplified repetitive business logic content by creating detailed businesslogic.md
- Updated documentation structure to follow rs-ai repository standards
🔧 API Documentation Corrections:
- Corrected API type from 'Transactional API' to 'Events API' throughout documentation
- Updated endpoint information to reflect actual implementation: https://api.bluecore.app/api/track/mobile/v1
- Fixed authentication method documentation (namespace token in payload vs Bearer header)
- Updated rate limiting information to reflect Events API (removed incorrect Transactional API limits)
- Updated References section with official Events API documentation link
📚 Documentation Structure Improvements:
- Created comprehensive businesslogic.md with detailed field mappings and transformation flows
- Streamlined main README by removing repetitive sections and adding cross-references
- Consolidated error handling, testing, and performance sections
- Updated RETL documentation for consistency
🔍 Code Analysis Verification:
- Verified endpoint match between code implementation and documentation
- Confirmed authentication method matches actual implementation
- Validated all API references against official Bluecore Events API documentation
Files modified:
- src/cdk/v2/destinations/bluecore/README.md (extensive updates)
- src/cdk/v2/destinations/bluecore/docs/businesslogic.md (created)
- src/cdk/v2/destinations/bluecore/docs/retl.md (updated)
The documentation now accurately reflects the actual Bluecore Events API implementation
and provides reliable information for users integrating with Bluecore through RudderStack.
* chore: docs
* chore: review comments
* fix: review comments
* fix: review comments
* fix: review comments
* chore: improve release check reliability
* fix: ensure the testing tool is cross-platform compatible (#4401)
* fix: make testing tool compatible with cross plateform
* fix: review comments
* fix: review comments
* chore: update klaviyo version to 2024-10-15, remove config restricted changes (#4387)
* chore: criteo audience api version upgrade (#4382)
* chore: criteo audience api version upgrade
* chore: add destination integration doc
* chore: resolving comments
* chore: groupEventsByType function optimization (#4398)
* fix: rt event ordering for hs (#4397)
* fix: router output ordering for tiktok_ads
* fix: test
* fix: test
* fix: test
* fix: rt event ordering for hs
* test: configure batch processing defaults for component tests
* fix: test folder structure for destinations (#4414)
* feat: update workflow to raise pr for dedicated transformer from operator (#4413)
* feat: update workflow to raise pr for dedicated transformer from operator
* Update .github/workflows/prepare-for-prod-dt-deploy.yml
Co-authored-by: Krishna Chaitanya <chaithu.kitti@gmail.com>
* chore: revert rollback changes
---------
Co-authored-by: Sudip Paul <sudip@rudderstack.com>
Co-authored-by: Krishna Chaitanya <chaithu.kitti@gmail.com>
* chore: remove broken workflow to updating dedicated deployment (#4418)
chore: remove broken updating dedicated deployment workflow
Co-authored-by: Sudip Paul <sudip@rudderstack.com>
* chore: update node version to 20.19.2 (#4416)
* chore(mailchimp): add router test with inputs contains different destnations (#4419)
* chore(mailchimp): add router test with inputs contains different destinations
* chore: misc
* chore(release): 1.103.0
* fix: enhance config object in ga4 v2 instead of mutating it (#4436)
* fix: enhance config object in ga4v2 instead of mutation
* chore: update changelog
---------
Co-authored-by: Utsab Chowdhury <utsab.c97@gmail.com>
Co-authored-by: Mahesh Kutty <39219085+maheshkutty@users.noreply.github.com>
Co-authored-by: Dilip Kola <33080863+koladilip@users.noreply.github.com>
Co-authored-by: Aris Tzoumas <atzoumas@rudderstack.com>
Co-authored-by: Vinay Teki <vinayteki95@gmail.com>
Co-authored-by: Sudip Paul <67197965+ItsSudip@users.noreply.github.com>
Co-authored-by: Dilip Kola <dilipkola@rudderstack.com>
Co-authored-by: Yashasvi Bajpai <33063622+yashasvibajpai@users.noreply.github.com>
Co-authored-by: Manish Kumar <144022547+manish339k@users.noreply.github.com>
Co-authored-by: Sudip Paul <sudip@rudderstack.com>
Co-authored-by: Krishna Chaitanya <chaithu.kitti@gmail.com>
Co-authored-by: GitHub Actions <noreply@github.com>
Co-authored-by: Sudip Paul <sudip@rudderstack.com>
* chore(test): add tests for onboarding postscript dest * chore: update tests, address comments * chore: some minor fixes in test fields * chore: some minor fixes in test fields
feat: upgrade Google Ads Remarketing Lists API from v18 to v19 - Updated @rudderstack/integrations-lib from 0.2.32 to 0.2.35 - Upgraded Google Ads Remarketing Lists API version from v18 to v19 - Updated API endpoints in config.js from v18 to v19 - Updated all test files to use v19 endpoints: - processor/data.ts - router/data.ts - dataDelivery/business.ts - dataDelivery/oauth.ts - network.ts - All 38 component tests passing with v19 API - No functional changes, only API version upgrade - Follows Linear ticket INT-3419 for Google APIs v19 upgrade
chore: remove errior logger from mailchimp Co-authored-by: Sudip Paul <sudip@rudderstack.com>
* fix: enhance config object in ga4v2 instead of mutation * chore: update changelog
* chore: update contributing.md to reflect latest guidelines * chore: addressing comments * chore: addressing comments
* chore: bump uuid from 9.0.1 to 11.1.0 Bumps [uuid](https://github.com/uuidjs/uuid) from 9.0.1 to 11.1.0. - [Release notes](https://github.com/uuidjs/uuid/releases) - [Changelog](https://github.com/uuidjs/uuid/blob/main/CHANGELOG.md) - [Commits](uuidjs/uuid@v9.0.1...v11.1.0) --- updated-dependencies: - dependency-name: uuid dependency-version: 11.1.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: fix commitlint issue --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sudip Paul <67197965+ItsSudip@users.noreply.github.com> Co-authored-by: Sudip Paul <sudip@rudderstack.com>
….1 in /.github/workflows (#4603) chore(deps): bump SonarSource/sonarqube-scan-action Bumps [SonarSource/sonarqube-scan-action](https://github.com/sonarsource/sonarqube-scan-action) from 5.3.0 to 5.3.1. - [Release notes](https://github.com/sonarsource/sonarqube-scan-action/releases) - [Commits](SonarSource/sonarqube-scan-action@v5.3.0...v5.3.1) --- updated-dependencies: - dependency-name: SonarSource/sonarqube-scan-action dependency-version: 5.3.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat: introduce endpointPath for iterable * chore: rename endpointLabel to endpointPath
Bumps [eslint](https://github.com/eslint/eslint) from 8.57.1 to 9.35.0. - [Release notes](https://github.com/eslint/eslint/releases) - [Changelog](https://github.com/eslint/eslint/blob/main/CHANGELOG.md) - [Commits](eslint/eslint@v8.57.1...v9.35.0) --- updated-dependencies: - dependency-name: eslint dependency-version: 9.35.0 dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4.4.0 to 5.0.0. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v4.4.0...v5.0.0) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@v7...v8) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 5.5.0 to 5.5.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v5.5.0...v5.5.1) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-version: 5.5.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4...v5) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
- Use UT_TAG_NAME instead of TAG_NAME for shared services - Fix yq path for hosted transformer image properties - Fix yq path for dedicated transformer enabled check - Use UT_TAG_NAME for dedicated transformer image version
* chore: update github action workflow actors * chore: lint fixes
- Update actions/checkout from v5 to v4 in build-pr-artifacts.yml - This ensures compatibility with older GitHub runners that don't support v5 - Addresses failing action executions due to runner version mismatches
…ormer into technocrats/ONCEHUB-103047-team
There was a problem hiding this comment.
Pull Request Overview
This PR updates the transformer version to 1.102.2 across the codebase. The changes primarily consist of adding comprehensive documentation for destination configurations, API endpoints, business logic, and mappings for multiple destinations, along with some bug fixes and feature improvements.
Key Changes:
- Update to transformer version 1.102.2
- Added extensive README.md files and documentation for multiple destinations
- Added business logic and RETL documentation for various destinations
- Bug fixes in network handlers, error handling, and utility functions
Reviewed Changes
Copilot reviewed 155 out of 432 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/v0/destinations/facebook_pixel/README.md |
Added comprehensive Facebook Pixel destination documentation |
src/v0/destinations/facebook_conversions/docs/retl.md |
Added RETL functionality documentation for Facebook Conversions |
src/v0/destinations/facebook_conversions/docs/businesslogic.md |
Added business logic and mappings documentation |
src/v0/destinations/facebook_conversions/config.js |
Added TODO comment for API version update |
src/v0/destinations/facebook_conversions/README.md |
Added comprehensive destination documentation |
| Plus 40+ additional documentation files and minor code improvements |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| // throw new Error( | ||
| // `[Unhandled Identify Resolution Error] ${brazeIdentifyResp.response?.errors[0]?.type}`, | ||
| // ); |
There was a problem hiding this comment.
[nitpick] Consider removing commented-out error throwing code. If this is needed for debugging, consider using a proper logging mechanism instead.
| // throw new Error( | |
| // `[Unhandled Identify Resolution Error] ${brazeIdentifyResp.response?.errors[0]?.type}`, | |
| // ); |
| return phoneNumberObject.format('E.164'); | ||
| } | ||
| throw new InstrumentationError('Invalid phone number'); | ||
| } catch (error) { |
There was a problem hiding this comment.
[nitpick] The catch block re-throws all errors including the intentional 'Invalid phone number' error from line 61, which creates redundant error wrapping. Consider checking if the error is already an InstrumentationError before wrapping it.
| } catch (error) { | |
| } catch (error) { | |
| if (error instanceof InstrumentationError) { | |
| throw error; | |
| } |
| (payload.clientUserId && payload.customIdentifiers) || | ||
| (hasOtherIdentifiers && payload.clientUserId) || | ||
| (hasOtherIdentifiers && payload.customIdentifiers); |
There was a problem hiding this comment.
[nitpick] The indentation for the multi-line boolean expression is inconsistent. Consider aligning the operators or using proper indentation for better readability.
| (payload.clientUserId && payload.customIdentifiers) || | |
| (hasOtherIdentifiers && payload.clientUserId) || | |
| (hasOtherIdentifiers && payload.customIdentifiers); | |
| (payload.clientUserId && payload.customIdentifiers) | |
| || (hasOtherIdentifiers && payload.clientUserId) | |
| || (hasOtherIdentifiers && payload.customIdentifiers); |
| parentPort.postMessage({ | ||
| type: MESSAGE_TYPES.AGGREGATE_METRICS_RES, | ||
| metrics: promString, | ||
| requestId: message.requestId, | ||
| }); | ||
| } catch (error) { | ||
| parentPort.postMessage({ type: MESSAGE_TYPES.AGGREGATE_METRICS_RES, error: error.message }); | ||
| parentPort.postMessage({ | ||
| type: MESSAGE_TYPES.AGGREGATE_METRICS_RES, | ||
| error: error.message, | ||
| requestId: message.requestId, | ||
| }); |
There was a problem hiding this comment.
[nitpick] The requestId is added to the response but there's no validation that message.requestId exists. Consider adding a check to ensure requestId is defined before including it in the response.
What are the changes introduced in this PR?
feat: [Team] ONCEHUB-98788: updating transformer version to 1.102.2
Write a brief explainer on your code changes.
feat: [Team] ONCEHUB-98788: updating transformer version to 1.102.2
What is the related Linear task?
Resolves INT-XXX
Please explain the objectives of your changes below
Put down any required details on the broader aspect of your changes. If there are any dependent changes, mandatorily mention them here
Any changes to existing capabilities/behaviour, mention the reason & what are the changes ?
N/A
Any new dependencies introduced with this change?
N/A
Any new generic utility introduced or modified. Please explain the changes.
N/A
Any technical or performance related pointers to consider with the change?
N/A
@coderabbitai review
Developer checklist
My code follows the style guidelines of this project
No breaking changes are being introduced.
All related docs linked with the PR?
All changes manually tested?
Any documentation changes needed with this change?
Is the PR limited to 10 file changes?
Is the PR limited to one linear task?
Are relevant unit and component test-cases added in new readability format?
Reviewer checklist
Is the type of change in the PR title appropriate as per the changes?
Verified that there are no credentials or confidential data exposed with the changes.