Add OSS-Realtime scan functionality to identify malicious packages (AST-95475, AST-95476, AST-95478) #1131
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduced a new `oss-realtime` subcommand for performing OSS-Realtime scans efficiently. This includes a mock response for testing, a command registration, and wrapper definitions to enable easy integration and extension. The scan command outputs results in JSON format.
Replaced hardcoded mock data with dynamic file source input for building mock scan results. Added validation for the source flag and improved error handling. This ensures greater flexibility and alignment with user-specified inputs.
|
New Issues (26)Checkmarx found the following issues in this Pull Request
|
Added functionality to perform OSS realtime scans, leveraging both HTTP and mock wrappers. Introduced local caching to optimize package scan results by avoiding redundant scans. Updated command dependencies and environment variable bindings to incorporate the new OSS Realtime Scanner logic.
Redefined the `ossrealtime` package to align with `commands`, and updated references accordingly. Introduced descriptive comments for `prepareScan` and `scanAndCache` functions to improve clarity. Included a new dependency (`github.com/bouk/monkey`) in `go.mod` and `go.sum`.
Introduce a `requirements.txt` file and `package.json` for dependency management, ensuring compatibility with OSS scanning. Add unit tests to verify successful and failed scans for supported and unsupported file types, improving command reliability.
… with random statuses
…ctor OSS-Realtime scan logic and add unit tests Renamed core functions for clarity, centralized cache key logic into the osscache package, and adjusted TTL handling. Updated CLI examples and added comprehensive tests for better scan and cache coverage.
Moved `Test_buildCacheMap` and `Test_cacheKey` to `osscache` package-specific tests to ensure proper encapsulation. Renamed associated functions to align with naming conventions (e.g., `BuildCacheMap`, `GenerateCacheKey`). Simplified `oss-realtime_test.go` by removing redundant cache-related tests.
Enhanced the sources flag to allow specifying multiple files separated by commas. Updated mock wrappers and tests to add support for caching and ensure proper handling of multiple file sources and cache usage scenarios.
Streamline the Go setup process by upgrading action versions and introducing cache cleaning and module fetching steps. Use tagged versions for actions instead of commit SHAs for readability and maintainability. Refine golangci-lint configuration to improve clarity and reliability.
Replaced deprecated and outdated linters with current alternatives, ensuring better alignment with modern practices. Adjusted linter settings, issue exclusions, and directory rules for improved linting precision and efficiency. Streamlined the configuration by removing redundant and legacy comments.
…update assertions for toScan package length
Introduced `OssRealtimeService` to encapsulate dependencies and methods related to OSSRealtime operations. Migrated functions from standalone implementations to methods of this service for improved modularity and clarity. Updated tests accordingly to use the new service structure.
cx-daniel-greenspan
previously approved these changes
May 19, 2025
cx-daniel-greenspan
previously approved these changes
May 19, 2025
cx-daniel-greenspan
approved these changes
May 19, 2025
cx-anjali-deore
pushed a commit
that referenced
this pull request
May 21, 2025
…ST-95475, AST-95476, AST-95478) (#1131) * Add OSS-Realtime scan functionality to identify malicious packages Introduced a new `oss-realtime` subcommand for performing OSS-Realtime scans efficiently. This includes a mock response for testing, a command registration, and wrapper definitions to enable easy integration and extension. The scan command outputs results in JSON format. * Refactor oss-realtime to use dynamic file source flag Replaced hardcoded mock data with dynamic file source input for building mock scan results. Added validation for the source flag and improved error handling. This ensures greater flexibility and alignment with user-specified inputs. * Integrate OSS Realtime Scanner with caching support Added functionality to perform OSS realtime scans, leveraging both HTTP and mock wrappers. Introduced local caching to optimize package scan results by avoiding redundant scans. Updated command dependencies and environment variable bindings to incorporate the new OSS Realtime Scanner logic. * Add realtimeScannerWrapper to integration test setup * Refactor OSS Realtime command structure and add comments Redefined the `ossrealtime` package to align with `commands`, and updated references accordingly. Introduced descriptive comments for `prepareScan` and `scanAndCache` functions to improve clarity. Included a new dependency (`github.com/bouk/monkey`) in `go.mod` and `go.sum`. * Add requirements.txt and package.json with tests for OSS scanning Introduce a `requirements.txt` file and `package.json` for dependency management, ensuring compatibility with OSS scanning. Add unit tests to verify successful and failed scans for supported and unsupported file types, improving command reliability. * Enhance RealtimeScannerMockWrapper to generate dynamic mock responses with random statuses * Refactor cache file path computation into a helper function. * Refactor OSS Realtime scan functions and enhance cache managementRefactor OSS-Realtime scan logic and add unit tests Renamed core functions for clarity, centralized cache key logic into the osscache package, and adjusted TTL handling. Updated CLI examples and added comprehensive tests for better scan and cache coverage. * Refactor OSS cache test logic and update function usage. Moved `Test_buildCacheMap` and `Test_cacheKey` to `osscache` package-specific tests to ensure proper encapsulation. Renamed associated functions to align with naming conventions (e.g., `BuildCacheMap`, `GenerateCacheKey`). Simplified `oss-realtime_test.go` by removing redundant cache-related tests. * Rename test function for OSS Realtime scan to improve clarity * Update scanning functionality to support multiple files input Enhanced the sources flag to allow specifying multiple files separated by commas. Updated mock wrappers and tests to add support for caching and ensure proper handling of multiple file sources and cache usage scenarios. * Update test file paths in OSS Realtime scan tests for consistency * Fix error message in ReadCache() test to dereference got value for clarity * Add additional dependencies for manifest parser in golangci configuration * Update manifest parser dependencies in golangci configuration * Update CI configuration for Go setup and linting Streamline the Go setup process by upgrading action versions and introducing cache cleaning and module fetching steps. Use tagged versions for actions instead of commit SHAs for readability and maintainability. Refine golangci-lint configuration to improve clarity and reliability. * Update golangci-lint version to v2.1.6 in CI configuration * Update golangci-lint action to a specific commit for consistency * Update golangci-lint configuration to version 2 and refine linter settings * Remove typecheck from golangci-lint configuration * Refactor golangci-lint configuration by removing deprecated settings and enabling essential linters * Enhance golangci-lint configuration with new linters and refined settings * Add 'go mod tidy' step to CI workflow for dependency management * Remove typecheck linter from golangci-lint configuration * Add typecheck linter and update golangci-lint version to 1.64.2 * Add typecheck linter and update golangci-lint version to 1.64.2 * fix linter errors * fix linter errors * Update file path in tests to correct relative directory Previously, the test referenced an incorrect relative path to `package.json`, causing potential failures when running tests. Updated the path to ensure it correctly points to the intended location. * Update file path in tests to reflect new directory structure * Update file paths in tests to match new directory structure * Replace reflect.DeepEqual with assert.Equal in cache tests * Update file path in tests to correct relative directory * Update assertion in cache test to use asserts.True for TTL comparison * Refactor OSS realtime scanning API and remove license check. Updated the OSS scanning API to use a single structured request object instead of slices of packages. Refined HTTP wrapper logic, adjusted mock implementations, and replaced license checks with a no-op for streamlined functionality. * Update CustomScan function to accept OssPackageRequest instead of slice * Fix linter * Refactor prepareScan and scanAndCache functions to use pointers for response and request types * Add integration tests and dependencies for OSS Realtime scans Introduced integration tests for OSS Realtime scans using `requirements.txt` and `package.json` files. Updated dependencies and modified the `scanAndCache` function to fix parameter handling in unit tests. These changes ensure better test coverage and functionality validation. * Refactor deleteCacheFile function to remove unnecessary parameter * Remove error logging for non-existent cache file in validation function * Update .golangci.yml to refine linting configuration Replaced deprecated and outdated linters with current alternatives, ensuring better alignment with modern practices. Adjusted linter settings, issue exclusions, and directory rules for improved linting precision and efficiency. Streamlined the configuration by removing redundant and legacy comments. * Remove shadow check from golangci.yml linting configuration * Remove revive linter from golangci.yml configuration * Update .golangci.yml to replace maligned with revive linter * Add nil check for JwtWrapper in ensureLicense function * Skip test for invalid license due to unimplemented license check and update assertions for toScan package length * Fix assertion to check length of Packages in toScan * Refactor OSSRealtime logic into a dedicated service Introduced `OssRealtimeService` to encapsulate dependencies and methods related to OSSRealtime operations. Migrated functions from standalone implementations to methods of this service for improved modularity and clarity. Updated tests accordingly to use the new service structure. * Remove unused RealtimeScannerWrapperParams struct from oss-realtime.go * Add error handling for missing file path in RunOssRealtimeScan * Add revive rule to golangci configuration to disable stuttering check * Add feature flag for OSS Realtime and update manifest-parser import path * Enable OSS Realtime feature flag in test cases * Refactor OSS Realtime scanning to use new data structures and improve error handling * Refactor OSS Realtime service methods to remove receiver and improve readability * Move OssPackage and OssPackageResults types to config.go for better organization * Refactor OSS Realtime scanning to create package map after preparing scan
cx-anjali-deore
pushed a commit
that referenced
this pull request
May 21, 2025
…ST-95475, AST-95476, AST-95478) (#1131) * Add OSS-Realtime scan functionality to identify malicious packages Introduced a new `oss-realtime` subcommand for performing OSS-Realtime scans efficiently. This includes a mock response for testing, a command registration, and wrapper definitions to enable easy integration and extension. The scan command outputs results in JSON format. * Refactor oss-realtime to use dynamic file source flag Replaced hardcoded mock data with dynamic file source input for building mock scan results. Added validation for the source flag and improved error handling. This ensures greater flexibility and alignment with user-specified inputs. * Integrate OSS Realtime Scanner with caching support Added functionality to perform OSS realtime scans, leveraging both HTTP and mock wrappers. Introduced local caching to optimize package scan results by avoiding redundant scans. Updated command dependencies and environment variable bindings to incorporate the new OSS Realtime Scanner logic. * Add realtimeScannerWrapper to integration test setup * Refactor OSS Realtime command structure and add comments Redefined the `ossrealtime` package to align with `commands`, and updated references accordingly. Introduced descriptive comments for `prepareScan` and `scanAndCache` functions to improve clarity. Included a new dependency (`github.com/bouk/monkey`) in `go.mod` and `go.sum`. * Add requirements.txt and package.json with tests for OSS scanning Introduce a `requirements.txt` file and `package.json` for dependency management, ensuring compatibility with OSS scanning. Add unit tests to verify successful and failed scans for supported and unsupported file types, improving command reliability. * Enhance RealtimeScannerMockWrapper to generate dynamic mock responses with random statuses * Refactor cache file path computation into a helper function. * Refactor OSS Realtime scan functions and enhance cache managementRefactor OSS-Realtime scan logic and add unit tests Renamed core functions for clarity, centralized cache key logic into the osscache package, and adjusted TTL handling. Updated CLI examples and added comprehensive tests for better scan and cache coverage. * Refactor OSS cache test logic and update function usage. Moved `Test_buildCacheMap` and `Test_cacheKey` to `osscache` package-specific tests to ensure proper encapsulation. Renamed associated functions to align with naming conventions (e.g., `BuildCacheMap`, `GenerateCacheKey`). Simplified `oss-realtime_test.go` by removing redundant cache-related tests. * Rename test function for OSS Realtime scan to improve clarity * Update scanning functionality to support multiple files input Enhanced the sources flag to allow specifying multiple files separated by commas. Updated mock wrappers and tests to add support for caching and ensure proper handling of multiple file sources and cache usage scenarios. * Update test file paths in OSS Realtime scan tests for consistency * Fix error message in ReadCache() test to dereference got value for clarity * Add additional dependencies for manifest parser in golangci configuration * Update manifest parser dependencies in golangci configuration * Update CI configuration for Go setup and linting Streamline the Go setup process by upgrading action versions and introducing cache cleaning and module fetching steps. Use tagged versions for actions instead of commit SHAs for readability and maintainability. Refine golangci-lint configuration to improve clarity and reliability. * Update golangci-lint version to v2.1.6 in CI configuration * Update golangci-lint action to a specific commit for consistency * Update golangci-lint configuration to version 2 and refine linter settings * Remove typecheck from golangci-lint configuration * Refactor golangci-lint configuration by removing deprecated settings and enabling essential linters * Enhance golangci-lint configuration with new linters and refined settings * Add 'go mod tidy' step to CI workflow for dependency management * Remove typecheck linter from golangci-lint configuration * Add typecheck linter and update golangci-lint version to 1.64.2 * Add typecheck linter and update golangci-lint version to 1.64.2 * fix linter errors * fix linter errors * Update file path in tests to correct relative directory Previously, the test referenced an incorrect relative path to `package.json`, causing potential failures when running tests. Updated the path to ensure it correctly points to the intended location. * Update file path in tests to reflect new directory structure * Update file paths in tests to match new directory structure * Replace reflect.DeepEqual with assert.Equal in cache tests * Update file path in tests to correct relative directory * Update assertion in cache test to use asserts.True for TTL comparison * Refactor OSS realtime scanning API and remove license check. Updated the OSS scanning API to use a single structured request object instead of slices of packages. Refined HTTP wrapper logic, adjusted mock implementations, and replaced license checks with a no-op for streamlined functionality. * Update CustomScan function to accept OssPackageRequest instead of slice * Fix linter * Refactor prepareScan and scanAndCache functions to use pointers for response and request types * Add integration tests and dependencies for OSS Realtime scans Introduced integration tests for OSS Realtime scans using `requirements.txt` and `package.json` files. Updated dependencies and modified the `scanAndCache` function to fix parameter handling in unit tests. These changes ensure better test coverage and functionality validation. * Refactor deleteCacheFile function to remove unnecessary parameter * Remove error logging for non-existent cache file in validation function * Update .golangci.yml to refine linting configuration Replaced deprecated and outdated linters with current alternatives, ensuring better alignment with modern practices. Adjusted linter settings, issue exclusions, and directory rules for improved linting precision and efficiency. Streamlined the configuration by removing redundant and legacy comments. * Remove shadow check from golangci.yml linting configuration * Remove revive linter from golangci.yml configuration * Update .golangci.yml to replace maligned with revive linter * Add nil check for JwtWrapper in ensureLicense function * Skip test for invalid license due to unimplemented license check and update assertions for toScan package length * Fix assertion to check length of Packages in toScan * Refactor OSSRealtime logic into a dedicated service Introduced `OssRealtimeService` to encapsulate dependencies and methods related to OSSRealtime operations. Migrated functions from standalone implementations to methods of this service for improved modularity and clarity. Updated tests accordingly to use the new service structure. * Remove unused RealtimeScannerWrapperParams struct from oss-realtime.go * Add error handling for missing file path in RunOssRealtimeScan * Add revive rule to golangci configuration to disable stuttering check * Add feature flag for OSS Realtime and update manifest-parser import path * Enable OSS Realtime feature flag in test cases * Refactor OSS Realtime scanning to use new data structures and improve error handling * Refactor OSS Realtime service methods to remove receiver and improve readability * Move OssPackage and OssPackageResults types to config.go for better organization * Refactor OSS Realtime scanning to create package map after preparing scan
cx-anjali-deore
pushed a commit
that referenced
this pull request
Jul 3, 2025
…ST-95475, AST-95476, AST-95478) (#1131) * Add OSS-Realtime scan functionality to identify malicious packages Introduced a new `oss-realtime` subcommand for performing OSS-Realtime scans efficiently. This includes a mock response for testing, a command registration, and wrapper definitions to enable easy integration and extension. The scan command outputs results in JSON format. * Refactor oss-realtime to use dynamic file source flag Replaced hardcoded mock data with dynamic file source input for building mock scan results. Added validation for the source flag and improved error handling. This ensures greater flexibility and alignment with user-specified inputs. * Integrate OSS Realtime Scanner with caching support Added functionality to perform OSS realtime scans, leveraging both HTTP and mock wrappers. Introduced local caching to optimize package scan results by avoiding redundant scans. Updated command dependencies and environment variable bindings to incorporate the new OSS Realtime Scanner logic. * Add realtimeScannerWrapper to integration test setup * Refactor OSS Realtime command structure and add comments Redefined the `ossrealtime` package to align with `commands`, and updated references accordingly. Introduced descriptive comments for `prepareScan` and `scanAndCache` functions to improve clarity. Included a new dependency (`github.com/bouk/monkey`) in `go.mod` and `go.sum`. * Add requirements.txt and package.json with tests for OSS scanning Introduce a `requirements.txt` file and `package.json` for dependency management, ensuring compatibility with OSS scanning. Add unit tests to verify successful and failed scans for supported and unsupported file types, improving command reliability. * Enhance RealtimeScannerMockWrapper to generate dynamic mock responses with random statuses * Refactor cache file path computation into a helper function. * Refactor OSS Realtime scan functions and enhance cache managementRefactor OSS-Realtime scan logic and add unit tests Renamed core functions for clarity, centralized cache key logic into the osscache package, and adjusted TTL handling. Updated CLI examples and added comprehensive tests for better scan and cache coverage. * Refactor OSS cache test logic and update function usage. Moved `Test_buildCacheMap` and `Test_cacheKey` to `osscache` package-specific tests to ensure proper encapsulation. Renamed associated functions to align with naming conventions (e.g., `BuildCacheMap`, `GenerateCacheKey`). Simplified `oss-realtime_test.go` by removing redundant cache-related tests. * Rename test function for OSS Realtime scan to improve clarity * Update scanning functionality to support multiple files input Enhanced the sources flag to allow specifying multiple files separated by commas. Updated mock wrappers and tests to add support for caching and ensure proper handling of multiple file sources and cache usage scenarios. * Update test file paths in OSS Realtime scan tests for consistency * Fix error message in ReadCache() test to dereference got value for clarity * Add additional dependencies for manifest parser in golangci configuration * Update manifest parser dependencies in golangci configuration * Update CI configuration for Go setup and linting Streamline the Go setup process by upgrading action versions and introducing cache cleaning and module fetching steps. Use tagged versions for actions instead of commit SHAs for readability and maintainability. Refine golangci-lint configuration to improve clarity and reliability. * Update golangci-lint version to v2.1.6 in CI configuration * Update golangci-lint action to a specific commit for consistency * Update golangci-lint configuration to version 2 and refine linter settings * Remove typecheck from golangci-lint configuration * Refactor golangci-lint configuration by removing deprecated settings and enabling essential linters * Enhance golangci-lint configuration with new linters and refined settings * Add 'go mod tidy' step to CI workflow for dependency management * Remove typecheck linter from golangci-lint configuration * Add typecheck linter and update golangci-lint version to 1.64.2 * Add typecheck linter and update golangci-lint version to 1.64.2 * fix linter errors * fix linter errors * Update file path in tests to correct relative directory Previously, the test referenced an incorrect relative path to `package.json`, causing potential failures when running tests. Updated the path to ensure it correctly points to the intended location. * Update file path in tests to reflect new directory structure * Update file paths in tests to match new directory structure * Replace reflect.DeepEqual with assert.Equal in cache tests * Update file path in tests to correct relative directory * Update assertion in cache test to use asserts.True for TTL comparison * Refactor OSS realtime scanning API and remove license check. Updated the OSS scanning API to use a single structured request object instead of slices of packages. Refined HTTP wrapper logic, adjusted mock implementations, and replaced license checks with a no-op for streamlined functionality. * Update CustomScan function to accept OssPackageRequest instead of slice * Fix linter * Refactor prepareScan and scanAndCache functions to use pointers for response and request types * Add integration tests and dependencies for OSS Realtime scans Introduced integration tests for OSS Realtime scans using `requirements.txt` and `package.json` files. Updated dependencies and modified the `scanAndCache` function to fix parameter handling in unit tests. These changes ensure better test coverage and functionality validation. * Refactor deleteCacheFile function to remove unnecessary parameter * Remove error logging for non-existent cache file in validation function * Update .golangci.yml to refine linting configuration Replaced deprecated and outdated linters with current alternatives, ensuring better alignment with modern practices. Adjusted linter settings, issue exclusions, and directory rules for improved linting precision and efficiency. Streamlined the configuration by removing redundant and legacy comments. * Remove shadow check from golangci.yml linting configuration * Remove revive linter from golangci.yml configuration * Update .golangci.yml to replace maligned with revive linter * Add nil check for JwtWrapper in ensureLicense function * Skip test for invalid license due to unimplemented license check and update assertions for toScan package length * Fix assertion to check length of Packages in toScan * Refactor OSSRealtime logic into a dedicated service Introduced `OssRealtimeService` to encapsulate dependencies and methods related to OSSRealtime operations. Migrated functions from standalone implementations to methods of this service for improved modularity and clarity. Updated tests accordingly to use the new service structure. * Remove unused RealtimeScannerWrapperParams struct from oss-realtime.go * Add error handling for missing file path in RunOssRealtimeScan * Add revive rule to golangci configuration to disable stuttering check * Add feature flag for OSS Realtime and update manifest-parser import path * Enable OSS Realtime feature flag in test cases * Refactor OSS Realtime scanning to use new data structures and improve error handling * Refactor OSS Realtime service methods to remove receiver and improve readability * Move OssPackage and OssPackageResults types to config.go for better organization * Refactor OSS Realtime scanning to create package map after preparing scan
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By submitting this pull request, you agree to the terms within the Checkmarx Code of Conduct. Please review the contributing guidelines for guidance on creating high-quality pull requests.
Description
Please provide a summary of the changes and the related issue. Include relevant motivation and context.
Type of Change
Related Issues
Link any related issues or tickets.
Checklist
Screenshots (if applicable)
Add screenshots to help explain your changes.
Additional Notes
Add any other relevant information.