Skip to content

Commit 109793c

Browse files
committed
Update documentation with latest fuzzers and arguments
1 parent ef451d6 commit 109793c

File tree

3 files changed

+17
-6
lines changed

3 files changed

+17
-6
lines changed

docs/commands-and-arguments/arguments.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,11 @@ You can get the full list of arguments by running `cats -h`. Below is a short de
5858
- `--ignoreResponseWords` COUNT_LIST a comma separated list of words count in the response that will be considered as SUCCESS, even if the Fuzzer will typically report it as WARN or ERROR
5959
- `--ignoreResponseLines` LINES_COUNT a comma separated list of lines count in the response that will be considered as SUCCESS, even if the Fuzzer will typically report it as WARN or ERROR
6060
- `--ignoreResponseRegex` a REGEX that will match against the response that will be considered as SUCCESS, even if the Fuzzer will typically report it as WARN or ERROR
61+
- `--filterResponseCodes` HTTP_CODES_LIST a comma separated list of HTTP response codes that will be filtered and not included in the final report. You can use response code families as `2xx`, `4xx`, etc.
62+
- `--filterResponseSize` SIZE_LIST a comma separated list of response sizes that will be filtered and not included in the final report
63+
- `--filterResponseWords` COUNT_LIST a comma separated list of words count in the response that will be filtered and not included in the final report
64+
- `--filterResponseLines` LINES_COUNT a comma separated list of lines count in the response that will be filtered and not included in the final report
65+
- `--filterResponseRegex` a REGEX that will match against the response that will be filtered and not included in the final report
6166
- `--tests` TESTS_LIST a comma separated list of executed tests in JSON format from the cats-report folder. If you supply the list without the .json extension CATS will search the test in the cats-report folder
6267
- `--ignoreResponseCodeUndocumentedCheck` If supplied (not value needed) it won't check if the response code received from the service matches the value expected by the fuzzer and will return the test result as SUCCESS instead of WARN
6368
- `--ignoreResponseBodyCheck` If supplied (no value needed) it won't check if the response body received from the service matches the schema supplied inside the contract and will return the test result as SUCCESS instead of WARN
@@ -91,10 +96,9 @@ You can get the full list of arguments by running `cats -h`. Below is a short de
9196
- `--fuzzersConfig=FILE` A properties file with Fuzzer configuration that changes default behaviour. Configuration keys are prefixed with the fully qualified Fuzzer name
9297
- `--mutators=FOLDER` A folder containing custom mutators. Only applicable when using the `cats random` sub-command
9398
- `--allowInvalidEnumValues` When set to true the `InvalidValuesInEnumsFieldsFuzzer` will expect a 2XX response code instead of 4XX
94-
- `--[no-]filterXxxFromRequestPayloads` In extremely rare cases when CATS fails to generate anyOf/oneOf combinations some requests may still contain ONE_OF/ANY_OF markers. They are filtered out by default. Setting this to false will send them as requests which will probably fail. It's mostly for debug purposes
95-
- `--generateXxxCombinationsForResponses` Generate anyOf/oneOf combinations also for response schemas. By default it creates one response payload with all possibilities
9699
- `--selfReferenceDepth=<selfReferenceDepth>` Max depth for objects having cyclic dependencies
97100
- `--limitXxxOfCombinations=<limitXxxOfCombinations>` Max number of anyOf/oneOf combinations
101+
- `--limitFuzzedFields=<numberOfFields>` Max number of fields that will be fuzzed
98102
- `--[no-]useDefaults` If set to true, it will use the default values when generating examples
99103
- `--nameReplace` If set to true, it will simply do a replacement between the targetFields names provided and the fuzz values
100104
- `--stopAfterErrors=<stopAfterErrors>` Number of errors after which the continuous fuzzing will stop running. Errors are defined as conditions matching the given match arguments. Only available in `cats random` sub-command.

docs/getting-started/filtering-reports.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ description: How to ignore specific HTTP response
77
CATS reports can be huge. Opening a report with 20k tests it's not the best user experience.
88
Outside the typical [Slicing Strategies](slicing-strategies), you can also choose to ignore reporting for specific HTTP responses.
99

10+
1011
## Ignoring Specific HTTP Responses
1112
By default, CATS will report `warns` and `errors` according to the specific behaviour of each Fuzzer. There are cases though when you might want to focus only on critical bugs.
1213
You can use the `--ignoreResponseXXX` arguments to supply a list of response codes, response sizes, word counts,
@@ -36,3 +37,7 @@ You can ignore specific response codes using the `--ignoreResponseCodes` argumen
3637

3738
## Ignore Response Lines, Words, Size, Regex
3839
You can ignore specific response lines, words, size or regex using the `--ignoreResponseLines`, `--ignoreResponseWords`, `--ignoreResponseSize`, `--ignoreResponseRegex` arguments. This will make CATS report these cases as `success` instead of `warn`.
40+
41+
:::info All --ignoreXXX arguments have a corresponding --filterXXX argument
42+
All `--ignoreXXX` arguments above have a corresponding `--filterXXX` argument which is equivalent to the `--ignoreXXX` argument AND will also consider `--skipReportingForIgnore` as being true.
43+
:::

docs/troubleshooting/index.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ These are some of the common reasons why CATS might appear to be stuck:
66

77
- **Network issues**: CATS might be waiting for a response from the server. Check the network connection and the server status.
88
- **Server issues**: The server might be slow or unresponsive. Check the server logs and the server status.
9-
- **Many fields inside the request(s)**: If the request has many fields, CATS might take longer to generate the tests, as many fuzzers are running for each field.
9+
- **Many fields inside the request(s)**: If the request has many fields, CATS might take longer to generate the tests, as many fuzzers are running for each field. You can limit the number of fields to be fuzzed using `--limitFuzzedFields` argument.
1010
- **Usage of anyOf/oneOf in request schemas**: If the request schema contains `anyOf` or `oneOf`, CATS will generate tests for all possible combinations. This can lead to a large number of tests being generated.
1111
You can limit the number of combinations by using the `--limitXxxCombinations` argument. For example, `--limitOneOfCombinations=10` will limit the number of combinations for all `oneOf/anyOf` to 10.
1212
- **Self-reference in the request schema**: If the request schema contains self-references, when CATS generates the request sample it might enter an infinite loop. You can limit the depth of the schema by using the `--selfReferenceDepth` argument. For example, `--selfReferenceDepth=3` will limit the self-reference depth of the schema to 3.
@@ -23,7 +23,7 @@ You can set examples for each property or provide specific formats in the schema
2323
CATS generates tests based on the request schema and the fuzzers that are enabled. If the request schema is complex or has many fields, CATS might generate a large number of tests.
2424
You can check the following pages for different strategies: [Slicing Strategies](https://endava.github.io/cats/docs/getting-started/slicing-strategies) and [Filtering Options](https://endava.github.io/cats/docs/getting-started/filtering-reports).
2525

26-
## CATS stops saying it cannot generate values for a field
26+
## CATS stops, saying it cannot generate values for a field
2727

2828
CATS generates values based on the request schema using examples or request field names and/or formats. If the request schema is not well-defined, CATS might not be able to generate values for some fields.
2929
There might be cases when the `pattern` for a specific property is too complicated and all internal generators are not able to create a value that it's matching that pattern.
@@ -35,7 +35,7 @@ This is usually when CATS generates a large number of tests and the report is to
3535

3636
## CATS generates same number of tests in 2 consecutive runs with same configuration
3737

38-
Unlike typical fuzzers that generate random values, CATS generates tests based on the request schema and the fuzzers that are enabled. If the request schema and the configuration are the same, CATS will generate the same tests.
38+
Unlike typical fuzzers that rely only on randomness, CATS generates tests based on the request schema and the fuzzers that are enabled. If the request schema and the configuration are the same, CATS will generate the same tests.
3939
CATS uses a deterministic approach to generate tests based on a set of fuzzers which run in a specific order and specific logic. Randomness is introduced only in the values generated for the fields.
4040
If you want randomness also in the tests generated, you must use the `cats random` sub-command. Check [Continuous Fuzzing](https://endava.github.io/cats/docs/getting-started/running-cats#continuous-fuzzing-mode) for more details.
4141

@@ -49,7 +49,7 @@ Assuming the OpenAPI file contains at least one valid path with at least one ope
4949
## CATS generates tests that are not valid
5050

5151
If you think some of the test cases generated are not valid in your scenario you can:
52-
- **Filter out the invalid test cases**: You can use the `--ignoreXXX` arguments to filter out the test cases that are not valid
52+
- **Filter out the invalid test cases**: You can use the `--ignoreXXX` or `--filterXXX` arguments to filter out the test cases that are not valid
5353
- **Filter out specific fuzzers**: You can use the `--skipFuzzer` argument to filter out specific fuzzers that are generating invalid test cases
5454
- **Filter out specific paths**: You can use the `--skpiPath` argument to filter out specific paths that are generating invalid test cases
5555
- **Filter out specific http methods**: You can use the `--skipHttpMethod` argument to filter out specific http methods that are generating invalid test cases
@@ -58,6 +58,8 @@ If you think some of the test cases generated are not valid in your scenario you
5858
- **Filter out specific OpenAPI field types**: You can use the `--skipFieldType` argument to filter out specific OpenAPI field types that are generating invalid test cases
5959
- **Filter out specific request headers**: You can use the `--skipHeader` argument to filter out specific request headers that are generating invalid test cases
6060

61+
If you think there might be a bug in CATS, please open an issue on the [GitHub repository](https://github.com/Endava/cats/issues/new/choose).
62+
6163
## CATS generates requests that are not valid
6264

6365
In very rare cases CATS might generate requests that are not valid.

0 commit comments

Comments
 (0)