You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Changes in file LICENSE.md:
* reworded advisory for readability
* related work
Changes in file docs/Testing.md:
* improved documentation regarding testing a bunch
* related work
1. Clone a [development build](https://github.com/reactive-firewall/multicast/tree/patch-readme-docs?tab=readme-ov-file#developer-builds)
65
+
66
+
1. Invoke the MATs test target.
67
+
68
+
```bash
69
+
make test-mats
70
+
```
71
+
72
+
#### Categories
73
+
74
+
Currently MATs are comprised of the test categories `bootstrap`, `basic`, `build`, `doctests`,
75
+
`say`, `hear`, and `usage`.
76
+
77
+
| Category | Criteria |_TestSuites_|
78
+
|----------|----------|--------------|
79
+
|`bootstrap`| Tests the basic function of initializations, exceptions, environment, and the `skt` component |[test_exceptions.ExceptionsTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_exceptions.py)|
80
+
|`basic`| Tests the trivial functions of initializations, exceptions, and environment |[test_basic.BasicTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_basic.py)|
81
+
|`build`| Tests the build and packaging support |[test_build.BuildPEP517TestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_build.py), [test_build.BuildPEP621TestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_build.py), [test_manifest.ManifestInclusionTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_manifest.py), ~[test_install_requires.ParseRequirementsTestSuite](https://github.com/reactive-firewall/multicast/blob/v2.0.8/tests/test_install_requires.py)~ (_removed in v2.0.9a3_) |
82
+
|`doctests`| Tests the doctests found in the `multicast` codebase |[multicast.__docs__](https://github.com/reactive-firewall/multicast/blob/HEAD/multicast)|
83
+
|`say`| Tests focused on `multicast/send.py`|[test_usage.MulticastTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_usage.py)|
84
+
|`hear`| Tests focused on `multicast/recv.py` and `multicast/hear.py`|[test_hear_server.McastServerTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_hear_server.py), [test_hear_server.HearUDPHandlerTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_hear_server.py), [test_hear_server_activate.McastServerActivateTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_hear_server_activate.py), [test_hear_data_processing.RecvDataProcessingTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_hear_data_processing.py), [test_hear_data_processing.HearHandleNoneDataTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_hear_data_processing.py), [test_hear_cleanup.HearCleanupTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_hear_cleanup.py)|
85
+
|`usage`| Tests focused on `multicast/__main__.py` and API use-cases |[test_usage.BasicIntegrationTestSuite](https://github.com/reactive-firewall/multicast/blob/HEAD/tests/test_usage.py)|
86
+
87
+
### Advanced Testing
88
+
89
+
> [!NOTE]
90
+
> # TL;DR - Contributor Focused
91
+
>
92
+
> The rest of this document is focused on topics that are intended only for those who wish to
93
+
> contribute (_or perhaps those few readers who are driven by an insatiable thirst for knowledge,
94
+
> and are here to delve into topics purely for the joy of learning_).
95
+
96
+
#### The Test Runner
97
+
98
+
The test runner is an often obscure part of the Multicast project testing process, albeit an
99
+
important part. In, general a test runner is exactly what it sounds like, the component responsible
100
+
for running the various tests during the act of testing.
101
+
102
+
> [!IMPORTANT]
103
+
> Both CI/CD and the Makefile ultimately use the same test runner under the hood.
104
+
105
+
```mermaid
106
+
graph TD;
107
+
CI["CI/CD"];
108
+
109
+
CI-->MAKE;
110
+
subgraph "Make test"
111
+
MAKE["Makefile"];
112
+
TOOL["Test Runner"];
113
+
TESTS["Multicast Project Tests"];
114
+
MAKE-->TOOL;
115
+
TOOL-->TESTS;
116
+
end
117
+
```
118
+
119
+
##### Contributor Context
120
+
121
+
Unlike the case of using `multicast` as a dependency for another project (e.g., the typical case),
122
+
contributors are instead modifying `multicast`.
123
+
49
124
Feature development and non-security related bug fixes are done on development branches and
50
125
then merged into the
51
126
[default branch (master)](https://github.com/reactive-firewall/multicast/blob/master/) for further
@@ -96,21 +171,229 @@ gitGraph:
96
171
commit id: "yet another small improvement"
97
172
```
98
173
99
-
### Linting
174
+
##### Test-Driven Contributors
175
+
176
+
During the development cycle new features will also need testing, (this is mentioned in detail
177
+
by the [Contributing Guidelines](https://github.com/reactive-firewall/multicast/tree/HEAD/.github/CONTRIBUTING.md).
178
+
This is where the other-wise undocumented Selective Test Runner [`run_selective.py`](https://github.com/reactive-firewall/multicast/tree/HEAD/tests/run_selective.py),
179
+
comes into play. While the test runner is used by much of the project automation to run all kinds
180
+
of tests, it is the most powerful when used to skip tests to save time iteratively developing a
181
+
new feature.
182
+
183
+
Consider the following special-cases where both CI/CD testing, and/or local `make test` testing
184
+
methods are unsuited:
185
+
186
+
* developing new features that yet to have their new tests implemented
187
+
* while a successful CI/CD (including coverage thresholds) is an important indicator that there
188
+
are no regressions, it does not provide feedback for the new features directly.
189
+
* only tests that regressed are ever of any actionable use
190
+
* and even tests that regressed may actually be false-positives, depending on the feature
191
+
* developing new tests
192
+
* checking more than just the new tests, until the new tests are passing, is inefficient.
193
+
194
+
These and similar cases can benefit from the improved granularity of targeted testing provided by
195
+
the selective test runner. Of-course eventually the process leads back to the normal CI/CD testing.
196
+
197
+
```mermaid
198
+
sequenceDiagram
199
+
actor user as Developer
200
+
participant Make as Makefile
201
+
participant CLI as Tests Runner
202
+
participant tests as Tests Module
203
+
204
+
alt via make
205
+
Note right of user: Typically user will use make to invoke the test runner
206
+
user->>Make: invoke a testing make target
207
+
Make-->Make: do Make things
208
+
Make->>CLI: invoke test runner
209
+
else via test runner
210
+
Note right of user: However, It is also entirely possible to directly invoke the test runner
211
+
user->>CLI: invoke test runner directly
212
+
end
213
+
CLI->>tests: run selected tests
214
+
loop Testing
215
+
tests->>tests: run test
216
+
end
217
+
tests->>CLI: done
218
+
CLI-->>user: Report Summary
219
+
opt if was via make
220
+
CLI->>Make: exited
221
+
Make-->Make: do Make things (e.g. Coverage Reporting)
222
+
Make->>user: done
223
+
end
224
+
```
225
+
226
+
#### Testing with Fuzzing
227
+
228
+
The Multicast Project includes some source-code to allow extra fuzzing when additional dependencies
229
+
are present.
230
+
231
+
> [!IMPORTANT]
232
+
> To effectively leverage the source code for performing fuzzing, you must install a licensed
233
+
> copy of the hypothesis python module.
234
+
235
+
There is a separate test group for optional fuzz testing aptly named `fuzzing`. Currently the
236
+
`fuzzing` test group contains only one category named `slow`.
237
+
238
+
| Category | Criteria |_TestSuites_|
239
+
|----------|----------|--------------|
240
+
|`slow`|_Optionally_ tests fuzzing `multicast` while ignoring test performance |[test_fuzz.HypothesisTestSuite](https://github.com/reactive-firewall/multicast/tree/HEAD/tests/test_fuzz.py)_Assuming it is present_|
241
+
242
+
#### Testing Performance
243
+
244
+
The Multicast Project includes some hooks for automating future performance testing.
245
+
246
+
> [!NOTE]
247
+
> Current performance testing is ad-hoc and focused on gathering more profiling data.
248
+
249
+
There is a separate test group intended for future performance testing aptly named `performance`.
250
+
Currently the `performance` test group contains only empty categories named `scalability`,
|`multi_sender`| Tests performance with multiple senders |_Empty_|
257
+
|`multi_receiver`| Tests performance with multiple listeners |_Empty_|
258
+
259
+
#### Testing even more
260
+
261
+
The Multicast Project includes some extra tests intended for additional, but non-acceptance,
262
+
testing (e.g., test for covering defensive or debugging code). These extra tests are organized
263
+
together under the aptly named group `extra`. The `extra` group contains the categories `coverage`,
264
+
`linting`, and `security`.
265
+
266
+
| Category |_Intended_ Criteria |_TestSuites_|
267
+
|----------|---------------------|--------------|
268
+
|`coverage`| Test for covering defensive or debugging code |[test_deps.BuildRequirementsTxtTestSuite](https://github.com/reactive-firewall/multicast/tree/HEAD/tests/test_deps.py), [test_hear_keyboard_interrupt.TestHearKeyboardInterrupt](https://github.com/reactive-firewall/multicast/tree/HEAD/tests/test_hear_keyboard_interrupt.py), [test_recv.McastRECVTestSuite](https://github.com/reactive-firewall/multicast/tree/HEAD/tests/test_recv.py)|
269
+
|`linting`| Tests for code structure and style |_Empty_|
270
+
|`security`| Tests related to secure coding practices |[test_extra.ExtraDocsUtilsTestSuite](https://github.com/reactive-firewall/multicast/tree/HEAD/tests/test_extra.py)|
271
+
272
+
##### Coverage
273
+
274
+
The Multicast project uses CI/CD to ensure constant and comprehensive testing. `coverage` in the
275
+
`extra` group's context is referring to "extra test coverage".
276
+
277
+
> [!TIP]
278
+
> Contributors who are unsure of how to categorize their new test code, may fall-back to the
279
+
> `coverage` category, so not to get bogged down before code review, where the categorization
280
+
> can better be discussed.
281
+
282
+
##### Linting
100
283
101
284
The Multicast project uses CI/CD to ensure all files, including code, configurations, and
102
285
documentation, are linted before publishing releases. This is automatically triggered by
103
286
pull requests during the project's release workflow, namely before merging anything to the
104
287
branch `master`, and again before releasing to `stable`.
105
288
106
-
#### Linting Badges
289
+
> [!WARNING]
290
+
Locally, linting is limited as many linters are copyleft restricted and can never be bundled with
make clean ;# cleans up from any previous tests hopefully
301
+
make test-style ;# runs the project's tests for style
302
+
make clean ;# cleans up for next test
303
+
```
304
+
305
+
#### Logging and Coverage testing
306
+
307
+
By default logging from the `multicast` module is ignored (there is a [default NULLHandeler](https://github.com/search?q=repo%3Areactive-firewall%2Fmulticast%20path%3Amulticast%2F*.py%20%22addHandler(logging.NullHandler())%22&type=code)
308
+
). The test-runner [`run_selective.py`](https://github.com/reactive-firewall/multicast/tree/HEAD/tests/run_selective.py)
309
+
inserts its own [special log handler](https://github.com/search?q=repo%3Areactive-firewall%2Fmulticast+path%3Atests%2F*.py+ColoredStreamHandler)
310
+
during testing to assist in debugging. Code coverage is only collected when invoking tests via the
311
+
[`Makefile`](https://github.com/reactive-firewall/multicast/tree/HEAD/Makefile) targets that
312
+
support it (and of-course only if already installed).
313
+
314
+
> [!TIP]
315
+
> Code coverage can leave a lot of artifacts after testing, so it is a good idea to run the
316
+
> `make clean` target when finished with each test run to blow-away such coverage artifacts.
317
+
318
+
Here is how `multicast` module logging and code coverage collection are integrated into the
319
+
whole testing process.
320
+
321
+
```mermaid
322
+
sequenceDiagram
323
+
actor user as Developer
324
+
participant Make as Makefile
325
+
participant CLI as Tests Runner
326
+
participant Python
327
+
participant tests as Tests Module
328
+
329
+
alt via make
330
+
user->>Make: invoke a testing make target
331
+
Make-->Make: do Make things
332
+
Make->>Python: check for coverage.py tool
333
+
alt with coverage.py installed
334
+
Python-->>Make: coverage.py
335
+
create participant COV as coverage.py
336
+
Make-->COV: start collecting
337
+
else without coverage.py
338
+
Python--xMake: Not found
339
+
end
340
+
Make->>CLI: invoke test runner
341
+
else via test runner
342
+
user->>CLI: invoke test runner directly
343
+
end
344
+
CLI->>Python: Initialize and Setup selected tests
345
+
create participant logging as `logging` module
346
+
Python-->>logging: initialize for test runner
347
+
Python-->>tests: Initialize and load selected tests
348
+
logging-->>Python: initialized
349
+
logging-->>CLI: `logger` object
350
+
tests-->>Python: initialized
351
+
tests-->>CLI: Setup and Loaded
352
+
CLI-->>user: starting
353
+
CLI->>tests: Run loaded tests
354
+
loop Testing
355
+
tests->>tests: run test
356
+
opt has output
357
+
tests-->>Python: logs any multicast module output
358
+
Python-->>logging: handle logging
359
+
logging->>CLI: log messages
360
+
CLI->>user: messages
361
+
end
362
+
alt passing
363
+
tests-->>CLI: Successful test
364
+
CLI-->>user: Reports test passed
365
+
else skipping
366
+
tests-->>CLI: will skip test
367
+
CLI-->>user: Reports skipped
368
+
else failing/error
369
+
tests--xCLI: Test failure
370
+
CLI-->>user: Reports Failed
371
+
end
372
+
end
373
+
tests->>CLI: done
374
+
CLI-->>user: Report Summary
375
+
CLI-xPython: cleanup
376
+
destroy logging
377
+
Python--xlogging: cleanup
378
+
Python--xtests: cleanup and exit
379
+
tests->>Python: done
380
+
Python->CLI: done
381
+
CLI-->>user: ending
382
+
CLI-xPython: exit
383
+
Python->>CLI: exited
384
+
opt if was via make
385
+
CLI->>Make: exited
386
+
alt with coverage.py installed
387
+
Make-->COV: stop collecting
388
+
destroy COV
389
+
Make--xCOV: save and report coverage results
390
+
end
391
+
Make-->Make: do Make things
392
+
Make->>user: done
393
+
end
394
+
```
112
395
113
-
### As a Dependency, Testing
396
+
####As a Dependency, Testing
114
397
115
398
> [!TIP]
116
399
> In a rush to get this module working? Then try using this in your own test workflow
0 commit comments