-
Notifications
You must be signed in to change notification settings - Fork 197
feat: implement scorecard-classic command #2433
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+2,227
−1
Merged
Changes from all commits
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
21d6c78
feat: implement new command scorecard-classic
AlbinaBlazhko17 978fc3d
chore: remove comments
AlbinaBlazhko17 9e17338
chore: add tests to meet unit test threshold
AlbinaBlazhko17 2eb091c
feat: add functionality to cache token
AlbinaBlazhko17 7579685
feat: implement functionality to fetch scorecard using api key
AlbinaBlazhko17 89ffb11
fix: import in fetchScorecard
AlbinaBlazhko17 3db9ed3
feat: implement functionality to output the scorecard results to JSON
AlbinaBlazhko17 d5fb893
fix: outdated tests
AlbinaBlazhko17 3602346
refactor: logger output
AlbinaBlazhko17 f55e773
feat: add tests
AlbinaBlazhko17 eefd800
chore: change url in tests
AlbinaBlazhko17 05da2d3
refactor: tests for scorecards
AlbinaBlazhko17 ad0af6a
fix: change output argument to format
AlbinaBlazhko17 5dee2b2
refactor: remove unused errors in catch and add warning log
AlbinaBlazhko17 107ce91
refactor: return statement in login handler
AlbinaBlazhko17 cffa873
fix: remove bundling before linting, change to resolveDocument
AlbinaBlazhko17 e24663f
feat: add verbose flag and detailed logs, add pluralize to logger mes…
AlbinaBlazhko17 b7cb6a1
feat: change tests due to changes and add tests for verbose flag
AlbinaBlazhko17 3ca9d3b
feat: add docs for new command
AlbinaBlazhko17 9b09ead
feat: add functionality to exit with code 1 if problems exist in openapi
AlbinaBlazhko17 4d1cb66
docs: add link how to configure scorecard
AlbinaBlazhko17 464d590
docs: fix errors to pass tests
AlbinaBlazhko17 2584eca
feat: add changeset
AlbinaBlazhko17 d94fd0f
chore: add log under verbose flag
AlbinaBlazhko17 dfa5a01
feat: add tests for login handler
AlbinaBlazhko17 0f9726f
feat: add fallback to old scorecard config
AlbinaBlazhko17 d66951b
fix: json formatter to match structure
AlbinaBlazhko17 7d1fdff
fix: logger info in stylish formatter
AlbinaBlazhko17 532c7ed
chore: add more obvious verbose message
AlbinaBlazhko17 b70b8f6
fix: tests after changes
AlbinaBlazhko17 0ffc518
feat: add functionality to check if it is a CI env and throw an error
AlbinaBlazhko17 3525480
fix: check for terminal interactivity
AlbinaBlazhko17 7eadbd4
feat: add functionality to check level of API
AlbinaBlazhko17 795a684
refactor: change params from positional to named
AlbinaBlazhko17 e45bbc3
fix: change logic to check if level has any problems
AlbinaBlazhko17 0f37952
feat: add tests for determineAchievedLevel function
AlbinaBlazhko17 6412586
Apply suggestions from code review
JLekawa db0ba06
fix: remove dupication in tests
AlbinaBlazhko17 0886f41
fix: types in command args
AlbinaBlazhko17 e7484d3
feat: add collecting spec data and error handling
AlbinaBlazhko17 c182102
fix: pass level if level has only warnings
AlbinaBlazhko17 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "@redocly/cli": patch | ||
| --- | ||
|
|
||
| Added `scorecard-classic` command to evaluate API descriptions against project scorecard configurations. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
| # `scorecard-classic` | ||
|
|
||
| ## Introduction | ||
|
|
||
| The `scorecard-classic` command evaluates your API descriptions against quality standards defined in your Redocly project's scorecard configuration. | ||
| Use this command to validate API quality and track compliance with organizational governance standards across multiple severity levels. | ||
|
|
||
| {% admonition type="info" name="Note" %} | ||
| The `scorecard-classic` command requires a scorecard configuration in your Redocly project. You can configure this in your project settings or by providing a `--project-url` flag. Learn more about [configuring scorecards](https://redocly.com/docs/realm/config/scorecard). | ||
| {% /admonition %} | ||
|
|
||
| ## Usage | ||
|
|
||
| ```bash | ||
| redocly scorecard-classic <api> --project-url=<url> | ||
| redocly scorecard-classic <api> --config=<path> | ||
| redocly scorecard-classic <api> --format=json | ||
| redocly scorecard-classic <api> --target-level=<level> | ||
| redocly scorecard-classic <api> --verbose | ||
| ``` | ||
|
|
||
| ## Options | ||
|
|
||
| | Option | Type | Description | | ||
| | -------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | api | string | Path to the API description filename or alias that you want to evaluate. See [the API section](#specify-api) for more details. | | ||
| | --config | string | Specify path to the [configuration file](#use-alternative-configuration-file). | | ||
| | --format | string | Format for the output.<br />**Possible values:** `stylish`, `json`. Default value is `stylish`. | | ||
| | --help | boolean | Show help. | | ||
| | --project-url | string | URL to the project scorecard configuration. Required if not configured in the Redocly configuration file. Example: `https://app.cloud.redocly.com/org/my-org/projects/my-project/scorecard-classic`. | | ||
| | --target-level | string | Target scorecard level to achieve. The command validates that the API meets this level and all preceding levels without errors. Exits with an error if the target level is not achieved. | | ||
| | --verbose, -v | boolean | Run the command in verbose mode to display additional information during execution. | | ||
|
|
||
| ## Examples | ||
|
|
||
| ### Specify API | ||
|
|
||
| You can use the `scorecard-classic` command with an OpenAPI description file path or an API alias defined in your Redocly configuration file. | ||
|
|
||
| ```bash | ||
| redocly scorecard-classic openapi/openapi.yaml --project-url=https://app.cloud.redocly.com/org/my-org/projects/my-project/scorecard-classic | ||
| ``` | ||
|
|
||
| In this example, `scorecard-classic` evaluates the specified API description against the scorecard rules defined in the provided project URL. | ||
|
|
||
| ### Use alternative configuration file | ||
|
|
||
| By default, the CLI tool looks for the [Redocly configuration file](../configuration/index.md) in the current working directory. | ||
| Use the optional `--config` argument to provide an alternative path to a configuration file. | ||
|
|
||
| ```bash | ||
| redocly scorecard-classic openapi/openapi.yaml --config=./another/directory/redocly.yaml | ||
| ``` | ||
|
|
||
| ### Configure scorecard in redocly.yaml | ||
|
|
||
| You can configure the scorecard project URL in your Redocly configuration file to avoid passing it as a command-line argument: | ||
|
|
||
| ```yaml | ||
| scorecard: | ||
| fromProjectUrl: https://app.cloud.redocly.com/org/my-org/projects/my-project/scorecard-classic | ||
|
|
||
| apis: | ||
| core@v1: | ||
| root: ./openapi/api-description.json | ||
| ``` | ||
|
|
||
| With this configuration, you can run the command without the `--project-url` flag: | ||
|
|
||
| ```bash | ||
| redocly scorecard-classic core@v1 | ||
| ``` | ||
|
|
||
| ### Use JSON output format | ||
|
|
||
| To generate machine-readable output suitable for CI/CD pipelines or further processing, use the JSON format: | ||
|
|
||
| ```bash | ||
| redocly scorecard-classic openapi/openapi.yaml --format=json | ||
| ``` | ||
|
|
||
| The JSON output is grouped by scorecard level and includes: | ||
|
|
||
| - version information | ||
| - achieved scorecard level | ||
| - summary of errors and warnings for each level | ||
| - rule ID and documentation link (for built-in rules) | ||
| - severity level (error or warning) | ||
| - location information (file path, line/column range, and JSON pointer) | ||
| - descriptive message about the violation | ||
|
|
||
| ### Validate against a target level | ||
|
|
||
| Use the `--target-level` option to ensure your API meets a specific quality level. The command validates that your API satisfies the target level and all preceding levels without errors: | ||
|
|
||
| ```bash | ||
| redocly scorecard-classic openapi/openapi.yaml --target-level=Gold | ||
| ``` | ||
|
|
||
| If the API doesn't meet the target level, the command: | ||
|
|
||
| - displays which level was actually achieved | ||
| - shows all validation issues preventing the target level from being met | ||
| - exits with a non-zero exit code (useful for CI/CD pipelines) | ||
|
|
||
| This is particularly useful in CI/CD pipelines to enforce minimum quality standards before deployment. | ||
|
|
||
| ### Run in verbose mode | ||
|
|
||
| For troubleshooting or detailed insights into the scorecard evaluation process, enable verbose mode: | ||
|
|
||
| ```bash | ||
| redocly scorecard-classic openapi/openapi.yaml --verbose | ||
| ``` | ||
|
|
||
| Verbose mode displays additional information such as: | ||
|
|
||
| - project URL being used | ||
| - authentication status | ||
| - detailed logging of the evaluation process | ||
|
|
||
| ## Authentication | ||
|
|
||
| The `scorecard-classic` command requires authentication to access your project's scorecard configuration. | ||
| You can authenticate in one of two ways: | ||
|
|
||
| ### Using API key (recommended for CI/CD) | ||
|
|
||
| Set the `REDOCLY_AUTHORIZATION` environment variable with your API key: | ||
|
|
||
| ```bash | ||
| export REDOCLY_AUTHORIZATION=your-api-key-here | ||
| redocly scorecard-classic openapi/openapi.yaml | ||
| ``` | ||
|
|
||
| ### Interactive login | ||
|
|
||
| If no API key is provided, the command prompts you to log in interactively: | ||
|
|
||
| ```bash | ||
| redocly scorecard-classic openapi/openapi.yaml | ||
| ``` | ||
|
|
||
| The CLI opens a browser window for you to authenticate with your Redocly account. | ||
|
|
||
| ## Scorecard results | ||
|
|
||
| The scorecard evaluation categorizes issues into multiple levels based on your project's configuration. | ||
| Each issue is associated with a specific scorecard level, allowing you to prioritize improvements. | ||
|
|
||
| The command displays the achieved scorecard level, which is the highest level your API meets without errors. | ||
| The achieved level is shown in both stylish and JSON output formats. | ||
|
|
||
| When all checks pass, the command displays a success message: | ||
|
|
||
| ```text | ||
| ☑️ Achieved Level: Gold | ||
|
|
||
| ✅ No issues found for openapi/openapi.yaml. Your API meets all scorecard requirements. | ||
| ``` | ||
|
|
||
| When issues are found, the output shows: | ||
|
|
||
| - the achieved scorecard level | ||
| - the rule that was violated | ||
| - the scorecard level of the rule | ||
| - the location in the API description where the issue occurs | ||
| - a descriptive message explaining the violation | ||
|
|
||
| If a `--target-level` is specified and not achieved, the command displays an error message and exits with a non-zero code. | ||
|
|
||
| ## Related commands | ||
|
|
||
| - [`lint`](./lint.md) - Standard linting for API descriptions with pass/fail results | ||
| - [`bundle`](./bundle.md) - Bundle multi-file API descriptions into a single file | ||
| - [`stats`](./stats.md) - Display statistics about your API description structure | ||
|
|
||
| ## Resources | ||
|
|
||
| - [API governance documentation](../api-standards.md) | ||
| - [Redocly configuration guide](../configuration/index.md) | ||
| - [Custom rules and plugins](../custom-plugins/index.md) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.