Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ elif [ "$1" = "format" ]; then
npm --prefix "$ROOT/src/hypofuzz/frontend" run format && shed
elif [ "$1" = "deps" ]; then
tox -e deps
elif [ "$1" = "deps-js" ]; then
npx npm-check-updates -u --packageFile src/hypofuzz/frontend/package.json && npm install --prefix src/hypofuzz/frontend
else
echo "Unknown build target $1. Available targets: docs, docs-clean, dashboard, dashboard-profiling, dashboard-json, dashboard-demo, test, format"
exit 1
Expand Down
13 changes: 12 additions & 1 deletion src/hypofuzz/collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
import pytest
from _pytest.nodes import Item
from _pytest.skipping import evaluate_condition
from hypothesis import Phase, settings
from hypothesis import HealthCheck, Phase, settings
from hypothesis.database import BackgroundWriteDatabase
from hypothesis.stateful import get_state_machine_test
from packaging import version
Expand Down Expand Up @@ -132,6 +132,17 @@ def pytest_collection_finish(self, session: pytest.Session) -> None:
)
continue

# nesting @given has undefined (for now) observability semantics,
# for example for PrimitiveProvider.on_observation. Skip until we
# can support it.
if HealthCheck.nested_given in test_settings.suppress_health_check:
self._skip_because(
"nested_given",
item.nodeid,
{"suppress_health_check": test_settings.suppress_health_check},
)
continue

# Wrap it up in a FuzzTarget and we're done!
try:
if hasattr(item.obj, "_hypothesis_state_machine_class"):
Expand Down
21 changes: 9 additions & 12 deletions src/hypofuzz/docs/index.rst
Original file line number Diff line number Diff line change
@@ -1,18 +1,15 @@
HypoFuzz documentation
======================
Welcome to HypoFuzz!
====================

:pypi:`Hypothesis` is free and open-source software, and always will be.
It's proven itself invaluable for everyone - from students to high-school
teachers, astrophysicists to data scientists, web developers to systems
programmers, and even for hardware designers.
HypoFuzz is an advanced fuzzing backend and dashboard for your `Hypothesis <https://github.com/HypothesisWorks/hypothesis//>`_ tests.

HypoFuzz builds on that success: if you have Hypothesis tests, HypoFuzz
gives you a fantastic way to spend CPU time - instead of engineering time -
to find bugs early in your development cycle.
Hundreds of thousands of developers trust Hypothesis to test their code, including :pypi:`NumPy <numpy>`, :pypi:`attrs`, `CPython <https://github.com/python/cpython>`_, :pypi:`pandas`, and :pypi:`cryptography`. HypoFuzz builds on top of the success of Hypothesis: HypoFuzz finds bugs quicker, gives greater insight into your test effectiveness, and makes more efficient use of CPU time—all using your same Hypothesis test suite.

* `HypoFuzz website <https://hypofuzz.com/>`_
* `HypoFuzz documentation <https://hypofuzz.com/docs/index.html>`_
* `HypoFuzz source code <https://github.com/Zac-HD/hypofuzz>`_
We recommend starting with the :doc:`quickstart <quickstart>`. For a full reference guide, check out the :doc:`user manual <manual/index>`.

* `Website <https://hypofuzz.com/>`_
* `Documentation <https://hypofuzz.com/docs/index.html>`_
* `Source code <https://github.com/Zac-HD/hypofuzz>`_

.. toctree::
:hidden:
Expand Down
13 changes: 7 additions & 6 deletions src/hypofuzz/docs/manual/collection.rst
Original file line number Diff line number Diff line change
@@ -1,16 +1,12 @@
Test collection
===============

Before HypoFuzz starts fuzzing by assigning tests to workers, it first needs to know what tests are available. HypoFuzz does so by running a collection step.
Before HypoFuzz starts fuzzing by assigning tests to workers, it first needs to know what tests are available. HypoFuzz does so by running a test collection step.

How HypoFuzz collects tests
---------------------------

HypoFuzz uses :pypi:`pytest` to collect test functions, with an almost identical command-line interface.

If you're using ``pytest ...`` to run your tests, ``hypothesis fuzz -- ...`` will fuzz them. Everything after ``--`` is passed to a call to ``pytest --collect-only`` in HypoFuzz.

Among others, this means you can use the standard pytest ``-k`` selector to configure which tests to run, with ``hypothesis fuzz -- -k ...``. See pytest's docs on :ref:`pytest:select-tests` and :ref:`pytest:test discovery` for more details.
HypoFuzz uses :pypi:`pytest` to collect test functions. Any arguments passed after ``hypothesis fuzz -- ...`` will be passed to pytest during collection. For example, this means you can use the standard pytest ``-k`` selector to configure which tests to run, with ``hypothesis fuzz -- -k ...``. See pytest's docs on :ref:`pytest:select-tests` and :ref:`pytest:test discovery` for more details.

Concretely, if your source layout looks like this:

Expand All @@ -28,6 +24,11 @@ and you normally run your tests with ``pytest``, then:
* ``hypothesis fuzz -- tests/test_a.py`` will fuzz the tests in ``test_a.py``
* ``hypothesis fuzz -- -k selector_string`` will fuzz the tests matching ``selector_string`` (see :ref:`pytest:select-tests`).

.. note::

In summary, if you're using ``pytest ...`` to run your tests, ``hypothesis fuzz -- ...`` will fuzz them.


Pytest plugins during collection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
1 change: 1 addition & 0 deletions src/hypofuzz/docs/manual/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ User manual
The user manual contains reference material on how to use HypoFuzz.

.. toctree::
:maxdepth: 2

operating
cli
Expand Down
10 changes: 5 additions & 5 deletions src/hypofuzz/docs/quickstart.rst
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
Quickstart
==========

This page will get you set up using HypoFuzz.
This quickstart will get you set up using HypoFuzz.

Prerequisite: Have some Hypothesis tests
----------------------------------------

HypoFuzz runs :pypi:`hypothesis` tests, so you'll need some of those first. If you don't have any Hypothesis tests yet, check out the :doc:`Hypothesis quickstart guide <hypothesis:quickstart>`, then come back after writing some!
HypoFuzz runs :pypi:`hypothesis` tests, so you'll need some of those first. If you don't have any Hypothesis tests yet, check out the :doc:`Hypothesis quickstart guide <hypothesis:quickstart>`, go write some Hypothesis tests, then come back!

Installation
------------
Expand All @@ -27,12 +27,12 @@ The no-argument command ``hypothesis fuzz`` does two things:
* Starts a local dashboard webserver, and
* Executes your Hypothesis tests using all available cores

These behaviors can be isolated with the ``--dashboard-only`` and ``--no-dashboard`` commands respectively, and the number of cores used can be controlled with ``-n/--num-processes``.
These behaviors can be isolated with the ``--dashboard-only`` and ``--no-dashboard`` commands respectively. The number of cores used can be controlled with ``-n/--num-processes``.

HypoFuzz uses :pypi:`pytest` to collect available tests. ``hypothesis fuzz`` should therefore be run in a directory where pytest can discover your tests.

Any arguments after ``hypothesis fuzz --`` are passed through to pytest. See :doc:`/manual/collection` for how to use this to configure which tests are collected.
Any arguments after ``hypothesis fuzz --`` are passed through to pytest. See :doc:`/manual/collection` for how to use this to control which tests are collected.

.. note::
.. seealso::

See the :doc:`command-line interface </manual/cli>` docs for a full command reference, and the :doc:`operating guide </manual/operating>` for advice on configuring HypoFuzz.
2 changes: 1 addition & 1 deletion src/hypofuzz/entrypoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ def _fuzz_impl(n_processes: int, pytest_args: tuple[str, ...]) -> None:
tests = collection.fuzz_targets
if not tests:
raise click.UsageError(
f"No property-based tests were collected. args: {pytest_args}"
f"No property-based tests were collected. pytest args: {pytest_args}"
)

skipped_s = "s" * (len(collection.not_collected) != 1)
Expand Down
Loading