diff --git a/docs/index.rst b/docs/index.rst index c804499e19..b2758f9155 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -10,6 +10,7 @@ - compile - contracts - testing + - contract_tests - reverts - networks - forking_networks diff --git a/docs/userguides/contract_tests.md b/docs/userguides/contract_tests.md new file mode 100644 index 0000000000..a8bce4cc55 --- /dev/null +++ b/docs/userguides/contract_tests.md @@ -0,0 +1,691 @@ +# Contract Tests + +Ape allows you to write "contract tests", which are test cases that are actually written +in a supported smart contract language, based on which compiler plugins you have installed. +They can be particularly useful when testing complex behaviors in your smart contracts, +such as dealing with reentrancy, doing property and invariant/stateful testing, and much more. + +## Test Suite Organization + +In order to write a smart contract test that will execute using our ape test runner +(see [Testing](./testing.html)), you will need to add smart contract files to your test suite +that follow a specific naming convention: + +``` +{config.test_folder}/ + .../ # NOTE: Can have any amount of subfolders in an Ape test suite + test*{.ext} + # `.ext` must be registered by supported compiler in your installed plugins +``` + +```{important} +Tests **MUST NOT** have multiple suffixes (e.g. `.t.sol`) in them to be registered. +This is done to avoid conflicts with other testing frameworks, +such as [Foundry](https://getfoundry.sh). +``` + +```{note} +It is recommended (but not required) that you title your test cases as snakecased names +(e.g. `test_something.sol`). +All a test needs to be registered is a filename prefixed with `test*`, +and an extension that matches a registered compiler plugin you have installed. +``` + +You can have any number of other contracts in these folders as well, +such as if you need special shared logic or utilities, +but note that these will not be registered as tests by our runner, +nor will they be compiled directly when running your test suite. + +## Writing Contract Tests + +Using the smart contract language of your choice, +you can write test cases in your contract test file by creating **ABI-exported methods** +that start with the name `test*`, for example: + +```solidity +function test_this_does_something() external { + // Testing logic here... +} +``` + +You can do anything you like inside these tests, you are only limited by the language itself! +Ape simply executes all of the registered test methods it finds in your contract test files, +and then shows any errors that raise from executing them. + +```{important} +Make sure that all of your test methods are "exported" into the compiled contract's ABI, +otherwise they will not register with our contract test runner, and therefore not be run. + +You can have as many internal functions as you want, just know those will not register either. +``` + +```{note} +You are free to make use of any code provided by project dependencies, +such as [`forge-std`](https://github.com/foundry-rs/forge-std). +These can give you access to language-specific testing features others have made for you, +such as [cheatcodes](https://getfoundry.sh/forge/tests/cheatcodes). + +Note that you will have to [configure any dependencies](../config.html#dependencies) +in your project's config first. +Also note that certain features (such as `forge-std`'s cheatcodes) will **only** work +on a supported node plugin (e.g. `anvil` w/ [`ape-foundry`](https://github.com/ApeWorX/ape-foundry)). + +Ape's contract test feature only makes miminal assumptions on how you write your tests, +and doesn't impose a particular plugin configuration to function. +``` + +### Pre-test Setup + +If your contract test module contains the special function `setUp`, +the test will be executed from a snapshot after deployment **that includes executing `setUp`**. +This mimics the same behavior from foundry where the initial state of a test starts after `setUp`. +However note that due to the snapshotting behavior, this feature is slightly more performant. + +### Accessing Fixtures from Ape + +One of Ape's best testing features (borrowed from [pytest](https://pytest.org)) +is ["fixtures"](https://docs.pytest.org/en/stable/how-to/fixtures.html), +which allows the configuration of shared, test-only parameters. +You'll be delighted to find that Contract Tests in Ape get fixtures for free! + +For example, let's say you have the following test fixture defined (in `tests/conftest.py`), +using the normal Ape syntax for working with your project's contracts: + +```py +@pytest.fixture +def token(project, TOTAL_SUPPLY, deployer): + return project.Token.deploy(TOTAL_SUPPLY, sender=deployer) +``` + +Then in your test, you can write the following to gain access to these (Python) fixture values: + +```solidity +function test_token_initialization( + IERC20 token, address deployer, uint256 TOTAL_SUPPLY +) external { + require(token.owner() == deployer); + require(token.totalSupply() == TOTAL_SUPPLY); + require(token.balanceOf(deployer) == TOTAL_SUPPLY); +} +``` + +This can **drastically** improve the speed of writing test cases! +And since fixtures are shared among all tests and obey all the same pytest rules, +you can leverage more advanced features like [fixture scoping][fixture-scoping], +[fixture parametrization][fixture-parametrization], and more! + +```{important} +Fixtures referenced in contract tests **MUST** be convertible to ABI arguments, +using Ape's conversion system. +If you reference a fixture that is **NOT** convertible to an ABI type when calling your test, +the test invocation will fail. + +Whatever langauge-specific internal type that argument has in your test (e.g. contract interfaces +vs. `address`) doesn't matter to Ape when invoking it, nor does the Python type of the value matter +(e.g. `"vitalik.eth"`) either, as long as it converts to the proper ABI-exported type for the arg. +``` + +### Natspec Test Modifiers + +Ape's contract test runner makes use of [Natspec](https://docs.soliditylang.org/en/v0.8.33/natspec-format.html) +(the "Ethereum Natural Langauge Specification" Format), and specifically "custom annotations", +in your test's contract- and function-level documentation strings in order to provide enhanced test +management features. This allows our test runner to pre-configure various settings and +[markers](https://docs.pytest.org/en/stable/how-to/mark.html) that control how your tests gets executed! + +This can be really useful for specifying things like "skip this test" +(via [`xfail`](https://docs.pytest.org/en/stable/how-to/skipping.html#xfail)), +or for leveraging more advanced features of pytest like "test parametrization" +(via [`parametrize`](https://docs.pytest.org/en/stable/how-to/parametrize.html#parametrizemark)). + +```{danger} +The arguments of several of these custom annotations are parsed with `eval` in order to obtain +their configured value in the testing context. + +**THIS CAN ALLOW ARBITRARY CODE EXECUTION**, so avoid running Contract Tests that you didn't +**personally write or review** to ensure that you don't execute improper code. + +See [the Python documentation](https://docs.python.org/3/library/functions.html#eval) for more +information about `eval` and it's potential dangers. +``` + +We support the following test Natspec modifiers (via custom annotations): + +#### Test Result Checkers + +_Used to check for specific **side effects** of running the test._ + +- `@custom:ape-check-reverts {expected error}` + + Check that running this test reverts with (the `eval` of) `{expected error}`. + + Example: + + ```solidity + /// @custom:ape-check-reverts "This error gets raised" + function test_reverts_with() external { + revert("This error gets raised"); + // NOTE: Test succeeds **only** if it reverted with "This error gets raised" + } + ``` + + ```{note} + `{expected error}` **MUST** be a string literal (like in our example), hex bytesvalue, + or it should `eval` in test context to a custom error type e.g. `my_contract.CustomError()`. + ``` + +- `@custom:ape-check-emits {expected logs...}` + + Check that after the test _succeeds_, the exact set of event logs in (the `eval` of) + `{expected logs...}` is emitted from running the test (must match **all** emitted logs). + + Example: + + ```solidity + /// @custom:ape-check-emits + /// - token.Approval(owner=self, spender=executor, value=100_000) + /// - token.Approval(spender=executor, value=10_000) + /// - token.Approval(owner=self, spender=executor) + /// - token.Approval(owner=self, value=100) + /// - token.Approval(value=10) + /// - token.Approval() + function test_emits(IERC20 token, address executor) external { + token.approve(executor, 100000); + token.approve(executor, 10000); + token.approve(executor, 1000); + token.approve(executor, 100); + token.approve(executor, 10); + token.approve(executor, 1); + } + ``` + + ```{caution} + Each case **MUST** use the `-` character prepended to the case to separate them. + ``` + + ```{note} + Similar to Ape, we can omit arguments in the mock log objects and it will match any value. + ``` + +#### Test Harness Setup + +_Configures how the test should be run._ + +- `@custom:ape-mark-xfail ` + + Run the test, but expect it to fail (and show `` as the failure reason for display). + +- `@custom:ape-mark-parametrize {cases...}` + + Create N cases of this test (where N is `len(cases)`), + where each case takes a particular set of values from (the `eval` of) `{cases...}` for `*args`. + + Example: + + ```solidity + /// @custom:ape-mark-parametrize investor,amount + /// - ("vitalik.eth", "100 ether") + /// - ("degen.eth", "10 ether") + /// - ("cowmilker.eth", "1 ether") + function test_token_initialization( + IERC20 token, address investor, uint256 amount + ) external { + require(token.balanceOf(investor) == amount); + } + ``` + + ```{caution} + Each case **MUST** use the `-` character prepended to the case to separate them. + ``` + + ```{important} + If only one parametrized argument is used, do **NOT** enclose each parametrized case in a tuple. + ``` + + ```{note} + This feature is similar to Foundry's "table tests" feature, but allows arbitrary python values. + ``` + +#### Property Test Settings + +_Configures the runner for [Property Testing](#property-testing)_ + +- `@custom:ape-fuzzer-max-examples {non-negative integer}` + + The maximum number of examples to generate for each Property Test. + Must be a non-negative integer (see [Hypothesis Documentation](https://hypothesis.readthedocs.io/en/latest/reference/api.html#hypothesis.settings.max_examples) for further information). + + ```{important} + When being used to control the maximum number of runs of a Stateful Test, this setting must be + set **at the contract-level**, not in individual `rule` function definitions. + ``` + + ```{note} + `max_examples` is similar to the `runs` configuration settings from Foundry. + ``` + +- `@custom:ape-fuzzer-deadline {milliseconds}` + + The deadline for each example to run for (in milliseconds). + Must be a non-negative integer (see [Hypothesis Documentation](https://hypothesis.readthedocs.io/en/latest/reference/api.html#hypothesis.settings.deadline) for further information). + +#### Stateful Test Settings + +_Configures the runner for [Stateful Testing](#stateful-testing)_ + +- `@custom:ape-stateful-step-count {non-negative integer}` + + The maximum number of `rule` calls to make in each Stateful Test example. + Must be a non-negative integer. + + ```{important} + This setting must be set **at the contract-level** to work, + not in individual `rule` function definitions. + ``` + + ```{note} + `max_examples` is similar to the `depth` configuration settings from Foundry. + ``` + +- `@custom:ape-stateful-bundles ` + + The list of "Bundles" (off-chain data collections that can be pulled for stateful test arguments) + that the test allows to use in a `rule`. + + To pull values from a Bundle in a test, simply use the name of the Bundle as the argument name: + + ```solidity + /// @custom:ape-stateful-bundles item_id + contract StatefulTest { + // ... + + function rule_use_bundle_item(item_id: uint256) external { + // Do something with `item_id`... + } + + // NOTE: When the Bundle name is an array of items, it pulls multiple + function rule_use_bundle_item(item_id: uint256[]) external { + // Do something with each `item_id`... + } + + // ... + } + ``` + + ```{important} + This setting must be set **at the contract-level** to work, + not in individual `rule` function definitions. + ``` + +- `@custom:ape-stateful-targets ` + + The Bundle to add the return value(s) of the resulting `rule` or `initialize` method, + to be accessed by subsequent `rule` invocations when the Bundle is used as an argument. + + To push values into a Bundle, do: + + ```solidity + /// @custom:ape-stateful-bundles item_id + contract StatefulTest { + // NOTE: When return value is an array of items, it pushes multiple items into the bundle + /// @custom:ape-stateful-targets item_id + function initialize_bundle_item() external returns (uint256[2]) { + // Adds `1` and `2` to the Bundle `item_id`, only once at start of test + return [uint256(1), uint256(2)]; + } + + /// @custom:ape-stateful-targets item_id + function rule_new_bundle_item() external returns (uint256) { + // Adds `3` to the Bundle `item_id`, each time rule is called + return 3; + } + + // ... + } + ``` + + ```{important} + This setting must be set **at the function-level** (of `initialize` or `rule` functions) to work, + not at the contract-level (where it will be ignored). + ``` + +- `@custom:ape-stateful-consumes ` + + Whether the Bundle argument should "consume" the value (e.g. remove the value from the Bundle), + when accessed by a `rule` invocation with that Bundle as an argument. + + To consume values from a Bundle, do: + + ```solidity + /// @custom:ape-stateful-bundles item_id + contract StatefulTest { + // ... + + // NOTE: The uint256 value is removed from `item_id` after executing this rule + /// @custom:ape-stateful-targets item_id + function rule_use_bundle_item(uint256 item_id) external { + // ... + } + + // ... + } + ``` + + ```{important} + This setting must be set **at the function-level** of `rule` functions to work, + not at the contract-level (where it will be ignored). + ``` + +### Property Testing + +"Property Tests" (also known as "Fuzz Tests" in Foundry) are tests that for the most part look like +normal tests, except that there are extra argument(s) given which do not match a known fixture (or +args in the `parametrizes` modifier). The presence of these extra argument(s) turn the single test +invocation in a series of invocations (configured by the `max_examples` +[Property Test modifier](#property-test-settings)) where the value for each extra argument is +pulled at random from a "Strategy", which describes the range of all possible values that type can +have. + +This adds an extra dimension to your testing, and often finds less obvious errors and bugs in your code, +so it is a good recommendation to add this to your project. +For example, say you have a "normal" contract test that looks like the following: + +```solidity +function test_minting_works( + IERC20 token, address executor +) external { + // NOTE: `executor` is the account executing our test + // (by default, same as `msg.sender`) + require(token.balanceOf(executor) == 0); + + // NOTE: The default "sender" for a mutable call is actually the test itself! + // (without using `vm.prank` mocking to impersonate a different account) + token.mint(executor, 1000) + require(token.balanceOf(executor) == 1000); +} +``` + +This test case demonstrates that a specific scenario works, +minting 1000 tokens to the `executor` address fixture leads to that account's balance increasing by 1000 tokens. + +To rewrite this as a Property Test, we would change `executor` to a new `address` variable named `acct` +(which doesn't match a fixture in our test suite), +and then add a parameter `amt` that will randomize the amount of tokens minted to `acct`. +Our new Property Test would look like: + +``` +// NOTE: This modifier is not necessary to make it a Property Test, but +// it is often useful to control the number of examples per test. +/// @custom:ape-fuzzer-max-examples 100 +function test_minting_works( + IERC20 token, address acct, uint256 amt +) external { + require(token.balanceOf(acct) == 0); + + token.mint(acct, amt) + require(token.balanceOf(acct) == amt); +} +``` + +Invoking this test might find an example where `acct` is not an allowed target for `token.mint`, +or `amt` is not an allowed amount of tokens to issue to `acct`. +This is kind of a contrived example, so it may yield pretty unexepected results, +however thanks to the power of Hypothesis and fixtures in Ape's Contract Testing feature, +we can actually **control the Strategy** of the values that we pull from (for `acct` and `amt`). + +Let use a fixture to define some custom strategies in our `conftest.py` for these variable names: + +```py +from eth_abi.tools import get_abi_strategy +from hypothesis import strategies as st + + +@pytest.fixture(scope="session") +def acct(investors): + # NOTE: Only get addresses that are **NOT** investors + return get_abi_strategy("address").filter(lambda a: a not in investors) + + +@pytest.fixture(scope="session") +def amt(TOTAL_SUPPLY): + # NOTE: Only select values for `amt` in test case, + # from the range `[0, TOTAL_SUPPLY]` + return st.integers(min_value=0, max_value=TOTAL_SUPPLY) +``` + +This can **drastically improve** the effectiveness of your Property Tests, as you can create +complex custom strategies that will find more relevant input scenarios more quickly! + +_See [Adapting Strategies](https://hypothesis.readthedocs.io/en/latest/tutorial/adapting-strategies.html) +and [Custom Strategies](https://hypothesis.readthedocs.io/en/latest/tutorial/custom-strategies.html) from +to learn more about strategies customization._ + +```{important} +Defining custom fuzzer strategies is more performant than the use of `vm.assume` cheatcode. +Because strategies define **the range of valid inputs** to pull from, they don't have to "reject" +invalid inputs that you wish to skip, because they are **never generated** in the first place! +``` + +```{note} +Hypothesis comes out of the box with support for "coverage expansion" behavior, +similar to Foundry's "coverage-guided fuzzing" concept. +Hypothesis's backend (which is [configurable](https://hypothesis.readthedocs.io/en/latest/extensions.html#alternative-backends) +for different scenarios like longer-term testing, SMT solving, etc.) will try and maximize the "coverage" of tests +(Property or Stateful) by picking new, unique inputs every time the test is executed. + +It will also pick previous examples that have been known to cause issues in the past, as well as "boundry conditions" +(values at the "boundry" of the strategy e.g. max and min value, zero, etc.) where it think it might find issues easily. +Selecting the "fuzzing domain" is best done through developing custom strategies for your tests, +and let the selection of the appropiate "value distributions" be up to Hypothesis's backend! +``` + +### Stateful Testing + +```{important} +Any "normal" tests (functions that start with `test`) in the test module will prevent registering +the module as a Stateful Test. + +The stateful test runner will **only** register a Stateful Test using the following `external` / +`public` methods with the below naming conventions (and associated state mutability): + +| function prefix | state mutability | +| :-------------: | :-----------------------: | +| `initialize` | `nonpayable` (default) | +| `rule` | `nonpayable` (default) | +| `invariant` | `view` / `pure` | +``` + +Sometimes, you have tests that require testing even more complex or interdependent behavior than +Property tests allow. Stateful Testing (also known as Invariant Testing) allows the generation of +complex scenarios where a randomized sequence of calls to different "rules" (functions that start +with `rule*`), each with potentially random arguments selected to be called with, can potentially +trigger more state-dependent logic than otherwise would get triggered by a single invocation by a +Property Test case. + +However, this requires a more complicated setup where a full test file is required to describe all +of the possibilities and rules for the test. For example, let's say we have a scenario where we +want to test the transfer logic of a token. We might want to explore what happens when we allow +different token holders to transfer tokens to each other. But to do that, we need a way to +introduce the concept of being a "holder" to our test, because if we just let the runner pick an +address at random (using the basic `address` Strategy) then the likelihood of selecting a holder +becomes astronomically small, leading to ineffective tests! + +#### Bundles + +Thankfully, Hypothesis's Stateful Test feature has the concept of "Bundles", which are buckets of +items that we can use in our Stateful Test as a strategy to select values from that matter to our +test. In this case, we want our `holder` Bundle to **only** contain addresses that **already** have +a balance. We can also use "initializers" (functions that start with `initialize*`) to pre-fill +Bundles so that our tests can start off accessing values from our Bundle which exist at the start +of the test (due to specific deployment or `setUp` logic). + +Such a case might look like: + +```solidity +// NOTE: Need this to access `vm` value from forge-std to use `prank` cheatcode +import {Test} from "forge-std/Test.sol"; + +// Creates a Bundle called `holder` to use inside the Stateful Test +/// @custom:ape-stateful-bundles holder +contract TokenTest is Test { + + // The return value(s) from this function will be added to the `holder` Bundle + /// @custom:ape-stateful-targets holder + function initialize_holders( + address deployer, address[] calldata investors + ) external returns (address[] memory) { + address[] memory initial_holders = new address[](1 + investors.length); + + // The token `deployer` gets an initial balance on deployment, so count as a holder + initial_holders[0] = deployer; + + // The `investors` have gotten an initial amount on deployment, so count them as holders + for (uint i = 0; i < investors.length; i++) + initial_holders[i + 1] = investors[i]; + + // This will "pre-fill" the `holder` Bundle with `initial_holders` (to pull in other tests) + return initial_holders; + } + + // The return value(s) from this function will be added to the `holder` Bundle too + /// @custom:ape-stateful-targets holder + function rule_transfer( + IERC20 token, address holder, address account, uint256 bips + ) external { + // NOTE: Get a portion of holder's balance by multiplying by our `bips` strategy + // (`bips` is a custom strategy, an integer that ranges from 0 to 10,000) + uint256 amount = token.balanceOf(holder) * bips / 10000; + + // Send that holder some of our tokens + vm.prank(holder); + token.transfer(account, amount); + + // NOTE: Adds `account` to Bundle `holder` (unless already in it) + return account; + } +} +``` + +Notice at the top, we use the `@custom:ape-stateful-bundles` modifier to create the Bundle +`holder` that we can fill with the current token holders during the test. +Then we wrote two functions that target this bundle (via `@custom:ape-stateful-targets`): +`initialize_holders`, which pre-fills this Bundle with the holders at the time of `token`'s +deployment from our fixtures, and `rule_transfer`, which sends a varying amount of tokens from +a `holder` to another address `account`, which then gets added to our Bundle. + +Thanks to the use of Bundles, we can now ensure that during our test, we only get **relevant** holders +as an input to our transfer rule, meaning we will no longer be randomly selecting 0 balance holders. +However you might notice a different problem, +which is what happens when a `holder` sends their **full balance** to `account`? +Well, unless you "consume" the value, the value remains in the Bundle for future use. +We can indicate this behavior through the use of `@custom:ape-stateful-consumes`: + +```solidity + // ... + + /// @custom:ape-stateful-consumes holder + /// @custom:ape-stateful-targets holder + function rule_transfer( + IERC20 token, address holder, address account, uint256 bips + ) external { + // ... + } + + // ... +``` + +By adding that we are "consuming" the value provided via `holder`, then each invocation of the rule +will remove the holder from the Bundle used by future invocations of the same rule. +But because we are also adding a new value to the bundle when the rule is executed, +we can keep the Bundle full of relevant values to select from for each step in the test! + +#### Invariants + +The last concept to understand about Stateful Testing is "invariants", +which are properties that must **always** hold during the entire execution of your test. +The default "invariant" that the runner checks for is the presence of any reverts that occur when invoking a `rule`. + +But lets say you actually have additional "invariants" you want to check **after every `rule` invocation**. +We can do that by defining extra `invariant*` view functions which are called to check on the internal state. + +This might look like: + +```solidity + // ... + + function invariant_check_total_supply( + IERC20 token, uint256 TOTAL_SUPPLY + ) view external { + require(token.totalSupply() == TOTAL_SUPPLY); + } + + // ... +``` + +For example, if during the course of operating our Stateful Test the value of `token.totalSupply()` +ever disagrees with `TOTAL_SUPPLY`, then it will show the sequnce of steps violating our invariant! + +You might imagine even more complex invariants that require stateful handling inside your test. +A common technique is to use "shadow variables", which are storage variables inside your test that +maintain values useful to checking your invariants, for instance "the sum of all `balanceOf` balances". + +Here's how you might employ a shadow variable in practice, first by initializing it during `setUp`, +then by checking inside one of your `invariant*` functions: + +```solidity + // ... + + uint256 balanceOf_sum; + + function setUp() { + balanceOf_sum = 0; + + // Other setup we need to do... + } + + // Initializers... (to set `holder` Bundle like above) + + /// @custom:ape-stateful-targets holder + function rule_mint( + MyToken token, address account, uint256 amount + ) external returns (address) { + token.mint(account, amount); + + // NOTE: Keep track of newly minted balance in shadow variable + balanceOf_sum += amount; + + // NOTE: Add `account` to `holder` Bundle. + return account; + } + + // Other rules... (`transfer`/`transferFrom` use, `burn`, etc.) + + function invariant_total_supply(MyToken token) view external { + require(token.totalSupply() == balanceOf_sum); + } + + // Other invariants... +``` + +_Stateful Testing is a complex subject, and it may benefit you to review the [Hypothesis Documentation](https://hypothesis.readthedocs.io/en/latest/stateful.html)._ + +```{note} +Due to their complex and multi-transaction flows, Stateful Tests can take longer to execute than +normal contract tests or even Property Tests. + +It is recommended to initially start by setting a small value for [`step_count`](#stateful-testing) +and [`max_examples`](#property-testing) while developing your test, and then only increasing those +limits once you like how the test is working (e.g. it runs without any intermittent failures). +``` + +## Running Contract Tests + +You can run contract tests alongside normal Python-based tests with the same Ape test runner, +and make use of all the same test discovery feature flags supported by `ape test`. +(see [`ape test` command userguide](./testing.html#ape-testing-commands)) + +The big difference with contract tests is that they are only compiled **when the test is run**, +so if there is a compilation error it will only raise during `ape test` (and not `ape compile`). + +[fixture-parametrization]: https://docs.pytest.org/en/stable/how-to/fixtures.html#parametrizing-fixtures +[fixture-scoping]: https://docs.pytest.org/en/stable/how-to/fixtures.html#scope-sharing-fixtures-across-classes-modules-packages-or-session diff --git a/docs/userguides/testing.md b/docs/userguides/testing.md index a2da06452e..875e316ca6 100644 --- a/docs/userguides/testing.md +++ b/docs/userguides/testing.md @@ -4,7 +4,8 @@ Testing an ape project is important and easy. ## Pytest -Before learning how testing works in Ape, you should have an understanding of [the pytest framework](https://docs.pytest.org/en/7.4.x/) and its concepts such as fixtures, mark-decorators, and pytest plugins such as x-dist, pytest-mock, and pytest-cov. +Before learning how testing works in Ape, you should have an understanding of the [pytest framework](https://pytest.org) +and its concepts such as fixtures, markers, and plugins (such as pytest-xdist, pytest-mock, and pytest-cov). Once you have learned about pytest, Ape testing becomes intuitive because it is built on top of pytest. In fact, `ape-test` is itself a `pytest` plugin! diff --git a/pyproject.toml b/pyproject.toml index e370e19d19..8c76579257 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -183,7 +183,7 @@ check_untyped_defs = true plugins = ["pydantic.mypy"] [tool.pytest.ini_options] -norecursedirs = "projects" +norecursedirs = ["projects", ".hypothesis"] # NOTE: 'no:ape_test' Prevents the ape plugin from activating on our tests # And 'pytest_ethereum' is not used and causes issues in some environments. diff --git a/src/ape/api/providers.py b/src/ape/api/providers.py index eca2b4a8d6..1dc396769c 100644 --- a/src/ape/api/providers.py +++ b/src/ape/api/providers.py @@ -105,7 +105,7 @@ def __repr__(self) -> str: return f"<{repr_str}>" @property - def datetime(self) -> datetime.datetime: + def datetime(self) -> "datetime.datetime": """ The block timestamp as a datetime object. """ diff --git a/src/ape/pytest/contracts/__init__.py b/src/ape/pytest/contracts/__init__.py new file mode 100644 index 0000000000..1fb88070aa --- /dev/null +++ b/src/ape/pytest/contracts/__init__.py @@ -0,0 +1,13 @@ +from .collector import ContractTestCollector +from .functional import ContractTestItem +from .module import ContractTestModule +from .stateful import StatefulTestItem +from .types import TestModifier + +__all__ = [ + ContractTestCollector.__name__, + ContractTestItem.__name__, + ContractTestModule.__name__, + StatefulTestItem.__name__, + TestModifier.__name__, +] diff --git a/src/ape/pytest/contracts/base.py b/src/ape/pytest/contracts/base.py new file mode 100644 index 0000000000..0fbbb9869d --- /dev/null +++ b/src/ape/pytest/contracts/base.py @@ -0,0 +1,127 @@ +from typing import TYPE_CHECKING, Any + +import pytest +from _pytest.fixtures import TopRequest + +from ape.utils import ManagerAccessMixin, cached_property + +from .types import TestModifier + +if TYPE_CHECKING: + from ethpm_types import ContractType + from ethpm_types.abi import ABIType + from hypothesis import settings as HypothesisSettings + + from ape.api.accounts import TestAccountAPI + from ape.contracts import ContractInstance + + +# TODO: Configure EVM context? Pre-compiles? Foundry-like cheatcodes? + + +class BaseTestItem(pytest.Item, ManagerAccessMixin): + def __init__( + self, + *, + name: str, + contract_type: "ContractType", + modifiers: dict[TestModifier, Any], + **kwargs, + ): + super().__init__(name=name, **kwargs) + + self.contract_type = contract_type + self.modifiers = modifiers + + if xfail_reason := self.modifiers.get(TestModifier.MARK_XFAIL): + self.add_marker(pytest.mark.xfail(reason=xfail_reason)) + + # TODO: Figure out a more "official" way to get fixtures by name + # HACK: Otherwise `.get_fixture_value` doesn't work + # NOTE: Copied this from pytest's own python test runner + fm = self.session._fixturemanager + fixtureinfo = fm.getfixtureinfo(node=self, func=None, cls=None) + self._fixtureinfo = fixtureinfo + self.fixturenames = fixtureinfo.names_closure + + def get_fixture_value(self, fixture_name: str) -> Any | None: + # NOTE: Use `_ispytest=True` to avoid `PytestDeprecationWarning` + # TODO: Refactor to `SubRequest` (avoid typing error below) + request = TopRequest(self, _ispytest=True) + + if fixture_defs := self.session._fixturemanager.getfixturedefs(fixture_name, self): + return fixture_defs[0].execute(request) # type: ignore[arg-type] + + return None + + @property + def hypothesis_settings(self) -> "HypothesisSettings": + settings_kwargs: dict = {} + + if max_examples := self.modifiers.get(TestModifier.FUZZ_MAX_EXAMPLES): + settings_kwargs["max_examples"] = max_examples + + if deadline := self.modifiers.get(TestModifier.FUZZ_DEADLINE): + settings_kwargs["deadline"] = deadline + + from hypothesis import settings + + return settings(**settings_kwargs) + + @property + def executor(self) -> "TestAccountAPI": + return self.account_manager.test_accounts[-1] + + @cached_property + def instance(self) -> "ContractInstance": + """ + The instance of `contract_type` to use for test case(s) found in contract test module. + + ```{important} + This will snapshot the instance's deployment so that each case can restart just after. + ``` + + ```{note} + If the module has `setUp` method, that will be called before snapshotting the instance. + ``` + """ + if hasattr(instance := self.executor.deploy(self.contract_type), "setUp"): + instance.setUp(sender=self.executor) + + self.snapshot = self.chain_manager.snapshot() + return instance + + @cached_property + def call_context(self) -> dict: + return { + # 1. Contract instance (document not to use bare storage or internal calls) + # Solidity instance (document not to use without `this.`) + "this": self.instance, + # Vyper instance + "self": self.instance, + # 2. Ape stuff + "msg": type( + "MsgContext", + (object,), + {"sender": self.executor}, + # TODO: Other parts of `msg.` context? + ), + # TODO: Other evm stuff? e.g. `tx`, `block`, etc. + } + + def get_value(self, abi_type: "ABIType") -> Any: + assert abi_type.name # mypy happy (always true) + if abi_type.name == "vm": + # NOTE: Foundry stdlib's VM instance + return "0x7109709ECfa91a80626fF3989D68f67F5b1DD12D" + + elif abi_type.name == "executor": + return self.executor + + elif fixture_value := self.get_fixture_value(abi_type.name): + return fixture_value + + # NOTE: Returning a Hypothesis strategy automatically converts to a fuzz tests + from eth_abi.tools import get_abi_strategy + + return get_abi_strategy(abi_type.canonical_type) diff --git a/src/ape/pytest/contracts/collector.py b/src/ape/pytest/contracts/collector.py new file mode 100644 index 0000000000..b6159c4f98 --- /dev/null +++ b/src/ape/pytest/contracts/collector.py @@ -0,0 +1,37 @@ +from collections.abc import Iterator +from typing import TYPE_CHECKING + +import pytest + +from ape.utils import ManagerAccessMixin + +if TYPE_CHECKING: + from .module import ContractTestModule + + +class ContractTestCollector(pytest.File, ManagerAccessMixin): + """Collect 1 (or more) Contract Tests from compiling file with a supported compiler.""" + + # TODO: `.compile_settings -> dict` to add test-only remappings + # TODO: Add `.local_project` to compiler settings + # TODO: Update Test-only compile config via `self.config_manager.get(...)`? + # TODO: Extend CompilerAPI plugin settings via `self.compiler.test_settings()` + # TODO: Support config settings via `.config` (from pytest config) + + def collect(self) -> Iterator["ContractTestModule"]: + if not (compiler := self.compiler_manager.registered_compilers.get(self.path.suffix)): + # TODO: Create a warning about missing compiler for extension? + return + + from .module import ContractTestModule + + for contract_type in compiler.compile( + [self.path], + # TODO: Use `settings=self.compile_settings` for test-only compile settings? + # Allows configuring extra test-only deps (e.g. `from ape.test import VM`) + ): + yield ContractTestModule.from_parent( + self, + name=contract_type.name, + contract_type=contract_type, + ) diff --git a/src/ape/pytest/contracts/functional.py b/src/ape/pytest/contracts/functional.py new file mode 100644 index 0000000000..e4c68b9bff --- /dev/null +++ b/src/ape/pytest/contracts/functional.py @@ -0,0 +1,93 @@ +from typing import TYPE_CHECKING, Any + +from hypothesis.strategies import SearchStrategy + +from ape.utils import cached_property +from hypothesis import given + +from ..contextmanagers import RevertsContextManager + +from .types import TestModifier +from .base import BaseTestItem + +if TYPE_CHECKING: + from ethpm_types.abi import MethodABI, ABIType + + from ape.contracts import ContractMethodHandler + + +class ContractTestItem(BaseTestItem): + def __init__( + self, + *, + abi: "MethodABI", + parametrized_args: dict | None = None, + **kwargs, + ): + super().__init__(**kwargs) + + self.abi = abi + self.parametrized_args = parametrized_args or {} + + @cached_property + def method(self) -> "ContractMethodHandler": + return getattr(self.instance, self.abi.name) + + def get_value(self, abi_type: "ABIType") -> Any: + # NOTE: Overrides BaseTestItem impl to also check parametrized case args + if parameterized_value := self.parametrized_args.get(abi_type.name): + return parameterized_value + + return super().get_value(abi_type) + + @cached_property + def call_args(self) -> dict[str, Any]: + """The args for calling the method in this specific case""" + + if any(ipt.name is None for ipt in self.abi.inputs): + raise RuntimeError(f"All input arguments in '{self}' must have a name.") + + return {ipt.name: self.get_value(ipt) for ipt in self.abi.inputs if ipt.name is not None} + + def eval_arg(self, raw_arg: str) -> Any: + # Just eval the whole string w/ global/local context from case + # NOTE: This is potentially dangerous, but only run on your own tests! + return eval(raw_arg, self.call_context, self.call_args) + + def runtest(self): + given_args: dict[str, SearchStrategy] = {} + for arg_name in (call_args := self.call_args): + if isinstance(arg := call_args[arg_name], SearchStrategy): + given_args[arg_name] = arg + call_args[arg_name] = None # NOTE: Placeholder for later update + + def test_case(**kwargs): + # NOTE: We need to retain ordering using the original dict + args = {k: v if v is not None else kwargs[k] for k, v in call_args.items()} + + if raw_revert_msg := self.modifiers.get(TestModifier.CHECK_REVERTS): + reverts_message = self.eval_arg(raw_revert_msg) + with RevertsContextManager(reverts_message): + self.method( + *args.values(), + sender=self.executor, + ) + + else: + # NOTE: Let revert bubble up naturally + receipt = self.method( + *args.values(), + sender=self.executor, + ) + + if raw_event_logs := self.modifiers.get(TestModifier.CHECK_EMITS): + expected_events = list(map(self.eval_arg, raw_event_logs)) + assert receipt.events == expected_events + + # TODO: Test reporting functionality? + + if given_args: + # NOTE: Re-write as a fuzzing case (leveraging Hypothesis integration) + test_case = given(**given_args)(self.hypothesis_settings(test_case)) + + test_case() diff --git a/src/ape/pytest/contracts/module.py b/src/ape/pytest/contracts/module.py new file mode 100644 index 0000000000..700f491035 --- /dev/null +++ b/src/ape/pytest/contracts/module.py @@ -0,0 +1,79 @@ +from typing import TYPE_CHECKING, Any + +import pytest + +from ape.utils import ManagerAccessMixin, cached_property + +from .types import TestModifier + +if TYPE_CHECKING: + from collections.abc import Iterator + + from ethpm_types import ContractType + from ethpm_types.abi import MethodABI + + from .base import BaseTestItem + + +class ContractTestModule(pytest.Collector, ManagerAccessMixin): + def __init__(self, *, contract_type: "ContractType", **kwargs): + super().__init__(**kwargs) + self.contract_type = contract_type + + @cached_property + def contract_modifiers(self) -> dict[TestModifier, Any]: + return TestModifier.parse_modifier_args(self.contract_type.devdoc) + + def get_method_modifiers(self, abi: "MethodABI") -> dict[TestModifier, Any]: + modifiers = TestModifier.parse_modifier_args( + self.contract_type.devdoc.get("methods", {}).get(abi.selector, {}) + ) + # NOTE: Cascade such that method-level overrides contract-level + modifiers.update({k: v for k, v in self.contract_modifiers.items() if k not in modifiers}) + return modifiers + + def collect(self) -> "Iterator[BaseTestItem]": + if any(abi.name.startswith("rule") for abi in self.contract_type.mutable_methods): + # It is a stateful test, so the whole module is a single item + from .stateful import StatefulTestItem + + yield StatefulTestItem.from_parent( + self, + name=self.name, + contract_type=self.contract_type, + modifiers=self.contract_modifiers, + ) + return + + from .functional import ContractTestItem + + # NOTE: Only mutable calls that have names starting with `test_` will work + for abi in self.contract_type.mutable_methods: + if abi.name.startswith("test"): + # 1. First parse the natspec for that test to obtain any `@custom:ape-*` modifiers + modifiers = self.get_method_modifiers(abi) + + # 2. Yield test cases (multiple if `mark.parametrize` exists) + if parametrized_args := modifiers.get(TestModifier.MARK_PARAMETRIZE): + # NOTE: If no cases collected, will not collect anything (fails silently) + for case_args in zip(*parametrized_args.values(), strict=True): + parametrized_str = "-".join(map(str, case_args)) + yield ContractTestItem.from_parent( + self, + name=f"{self.name}.{abi.name}[{parametrized_str}]", + contract_type=self.contract_type, + modifiers=modifiers, + parametrized_args=dict( + zip(parametrized_args.keys(), case_args, strict=True) + ), + abi=abi, + ) + + else: + yield ContractTestItem.from_parent( + self, + name=f"{self.name}.{abi.name}", + contract_type=self.contract_type, + modifiers=modifiers, + abi=abi, + ) diff --git a/src/ape/pytest/contracts/stateful.py b/src/ape/pytest/contracts/stateful.py new file mode 100644 index 0000000000..ef7887abf7 --- /dev/null +++ b/src/ape/pytest/contracts/stateful.py @@ -0,0 +1,179 @@ +from collections.abc import Callable +from types import new_class +from typing import TYPE_CHECKING, Any + +from ape.utils import cached_property +from eth_abi.tools import get_abi_strategy +from hypothesis import strategies as st +from hypothesis.stateful import ( + Bundle, + Rule, + RuleBasedStateMachine, + consumes, + initialize, + invariant, + multiple, + rule, + run_state_machine_as_test, +) + +from .base import BaseTestItem +from .types import TestModifier + +if TYPE_CHECKING: + from ethpm_types.abi import MethodABI + + +class StatefulTestItem(BaseTestItem): + def get_method_modifiers(self, abi: "MethodABI") -> dict[TestModifier, Any]: + modifiers = TestModifier.parse_modifier_args( + self.contract_type.devdoc.get("methods", {}).get(abi.selector, {}) + ) + # NOTE: Cascade such that method-level overrides contract-level + modifiers.update({k: v for k, v in self.modifiers.items() if k not in modifiers}) + return modifiers + + @cached_property + def bundles(self) -> dict[str, Bundle]: + # Check contract-level Natspec for bundle definitions + if not (names := self.modifiers.get(TestModifier.STATEFUL_BUNDLES)): + return {} + + return {name: Bundle(name) for name in names} + + def get_target(self, abi: "MethodABI") -> Bundle | None: + if not (modifiers := self.get_method_modifiers(abi)): + return None + + elif not (bundle_name := modifiers.get(TestModifier.STATEFUL_TARGETS)): + return None + + elif len(abi.outputs) != 1: + raise AssertionError( + f"'{self.path}:{abi.name}' must return exactly 1 value for bundle." + ) + + elif (target := self.bundles.get(bundle_name)) is None: + raise AssertionError( + f"'{self.path}:{abi.name}' has unrecognized bundle: '{bundle_name}'." + ) + + return target + + def consumes(self, abi: "MethodABI") -> set[str]: + if not (modifiers := self.get_method_modifiers(abi)): + return set() + + elif not (bundle_names := modifiers.get(TestModifier.STATEFUL_CONSUMES)): + return set() + + elif unrecognized_bundles := "', '".join(bundle_names - set(self.bundles)): + raise AssertionError( + f"'{self.path}:{abi.name}' has unrecognized bundle(s): '{unrecognized_bundles}'." + ) + + elif unrecognized_args := "', '".join(bundle_names - set(ipt.name for ipt in abi.inputs)): + raise AssertionError( + f"'{self.path}:{abi.name}' arg(s) reference unknown bundle(s): '{unrecognized_args}'." + ) + + return bundle_names + + def call_method(self, abi: "MethodABI") -> Callable: + if abi.stateMutability == "nonpayable": + + def wrapped_method(_: RuleBasedStateMachine, **kwargs): + method = self.instance._mutable_methods_[abi.name] + # TODO: Do we do proper lookup by name for value location? + # TODO: How to handle providing other txn_kwargs like `value=`? + receipt = method(*kwargs.values(), sender=self.executor) + + if isinstance(result := receipt.return_value, list): + return multiple(*result) + + # NOTE: Avoid returning empty tuple, when `None` expected + return result or None + + else: # view/pure (e.g. `invariant`) + + def wrapped_method(_: RuleBasedStateMachine, **kwargs): + method = self.instance._view_methods_[abi.name] + if isinstance(result := method(*kwargs.values()), list): + return multiple(*result) + + # NOTE: Avoid returning empty tuple, when `None` expected + return result or None + + wrapped_method.__name__ = abi.name + return wrapped_method + + @cached_property + def initializers(self) -> dict[str, Callable]: + initializers: dict[str, Callable] = {} + for abi in self.contract_type.mutable_methods: + if abi.name.startswith("initialize"): + if (target := self.get_target(abi)) is None: + raise AssertionError( + f"'{self.path}:{abi.name}' needs to target a bundle w/ `@custom:ape-stateful-targets`" + ) + + initializers[abi.name] = initialize(target=target)(self.call_method(abi)) + + return initializers + + @cached_property + def rules(self) -> dict[str, Rule]: + # TODO: Support preconditions? + rules: dict[str, Rule] = {} + for abi in self.contract_type.mutable_methods: + if abi.name.startswith("rule"): + decorator_args = {} + for ipt in abi.inputs: + if (bundle := self.bundles.get(ipt.name)) is not None: + if ipt.name in self.consumes(abi): + bundle = consumes(bundle) + + if "[" in ipt.canonical_type: + # TODO: Figure out how to specify max array size for vyper + bundle = st.lists(bundle, max_size=10) + + decorator_args[ipt.name] = bundle + + else: + decorator_args[ipt.name] = get_abi_strategy(ipt.canonical_type) + + if (target := self.get_target(abi)) is not None: + decorator_args["target"] = target + + rules[abi.name] = rule(**decorator_args)(self.call_method(abi)) + + return rules + + @cached_property + def invariants(self) -> dict[str, Callable]: + # TODO: Support preconditions? + return { + abi.name: invariant()(self.call_method(abi)) + for abi in self.contract_type.view_methods + if abi.name.startswith("invariant") + } + + @cached_property + def state_machine(self) -> type[RuleBasedStateMachine]: + def add_fields(ns): + ns.update(self.bundles) + ns.update(self.initializers) + ns.update(self.rules) + ns.update(self.invariants) + + return new_class( + self.name, + (RuleBasedStateMachine,), + exec_body=add_fields, + ) + + def runtest(self): + run_state_machine_as_test( + self.state_machine, + settings=self.hypothesis_settings, + ) diff --git a/src/ape/pytest/contracts/types.py b/src/ape/pytest/contracts/types.py new file mode 100644 index 0000000000..0473af183e --- /dev/null +++ b/src/ape/pytest/contracts/types.py @@ -0,0 +1,127 @@ +from enum import Enum +from typing import Any + + +class TestModifier(str, Enum): + """ + Enum that represents custom natspec annotations supported by Ape's Contract Test feature. + + These will be automatically parsed into `ContractTestModule.modifiers` + and `BaseContractTest.modifiers`, for use in modifying test handling. + """ + + # Test result checking modifiers + CHECK_REVERTS = "custom:ape-check-reverts" + CHECK_EMITS = "custom:ape-check-emits" + + # Test harness setup modifiers + MARK_PARAMETRIZE = "custom:ape-mark-parametrize" + MARK_XFAIL = "custom:ape-mark-xfail" + + # TODO: Support others? + # TODO: Fork testing? Marker for using different networks? + # (e.g. `@custom:ape-mark-fork ethereum:mainnet`) + + # Fuzz harness setup modifiers + FUZZ_MAX_EXAMPLES = "custom:ape-fuzzer-max-examples" + FUZZ_DEADLINE = "custom:ape-fuzzer-deadline" + + # Stateful harness setup modifiers + STATEFUL_STEP_COUNT = "custom:ape-stateful-step-count" + STATEFUL_BUNDLES = "custom:ape-stateful-bundles" + STATEFUL_TARGETS = "custom:ape-stateful-targets" + STATEFUL_CONSUMES = "custom:ape-stateful-consumes" + + @classmethod + def parse_modifier_args(cls, natspecs: dict) -> dict["TestModifier", Any]: + """Return the mapping of (supported) custom modifiers to their parsed args""" + + modifiers: dict[TestModifier, Any] = {} + for natspec in natspecs: + if not natspec.startswith("custom:"): + continue + + try: + modifier = cls(natspec) + + except Exception as e: + from ape.logging import logger + + all_modifiers = "', '".join(e.value for e in cls) + logger.warn_from_exception( + e, f"Unknown modifier {natspec}. Must be one of '{all_modifiers}'." + ) + continue + + modifiers[modifier] = modifier.parse_args(natspecs[natspec]) + + return modifiers + + def __str__(self) -> str: + return self.value + + def _split_args(self, raw_args: str) -> list[str]: + # Examples: + # 1. Only one arg on same line: "..." + # @custom:ape-check-reverts ... + # 2. No arg on same line, but multiple after: "- ... - ..." + # @custom:ape-check-emits + # - ... + # - ... + # 3. Arg on same line, and multiple after: "... - ... - ..." + # @custom:ape-mark-parametrize ... + # - ... + # - ... + # TODO: Does `Solidity` parse them this same way? Should it be per compiler plugin? + + if not raw_args: + return [] + + # NOTE: Do `.lstrip("-")` on `raw_args` to remove first instance of `-` + # **in scenarios where we don't have an arg on the same line first**. + return [ln.strip() for ln in raw_args.lstrip("-").split("-")] + + def parse_args(self, args: str) -> Any: + match self: + case TestModifier.CHECK_REVERTS: + return args + + case TestModifier.CHECK_EMITS: + return self._split_args(args) + + case TestModifier.MARK_PARAMETRIZE: + parameters, *raw_cases = self._split_args(args) + case_tuples = [eval(a, {}, {}) for a in raw_cases] + if not any(isinstance(i, tuple) for i in case_tuples): + assert "," not in parameters + return {parameters: case_tuples} + + return dict( + zip( + parameters.split(","), + # NOTE: Must be context-independent values to eval + list(zip(*case_tuples, strict=True)), + strict=True, + ) + ) + + case TestModifier.MARK_XFAIL: + return args + + case TestModifier.FUZZ_MAX_EXAMPLES: + return int(args) + + case TestModifier.FUZZ_DEADLINE: + return int(args) + + case TestModifier.STATEFUL_STEP_COUNT: + return int(args) + + case TestModifier.STATEFUL_BUNDLES: + return args.split(" ") + + case TestModifier.STATEFUL_CONSUMES: + return set(args.split(" ")) + + case TestModifier.STATEFUL_TARGETS: + return args diff --git a/src/ape/pytest/plugin.py b/src/ape/pytest/plugin.py index 1df52ab4f7..0ace99fb0b 100644 --- a/src/ape/pytest/plugin.py +++ b/src/ape/pytest/plugin.py @@ -1,8 +1,13 @@ import sys from pathlib import Path +from typing import TYPE_CHECKING + from ape.exceptions import ConfigError +if TYPE_CHECKING: + from pytest import Collector + def pytest_addoption(parser): def add_option(*names, **kwargs): @@ -116,3 +121,23 @@ def is_module(v): config.addinivalue_line( "markers", "use_network(choice): Run this test using the given network choice." ) + + +# NOTE: Below is done to support contract-based compiled tests +def pytest_collect_file(parent, file_path) -> "Collector | None": + # NOTE: Skip common files we know should not be used with this collector + if file_path.suffix in (".py", ".md", ".json"): + return None + + # NOTE: Avoid capturing "foundry tests", which follow a paradigm of `.t.sol` + elif len(file_path.suffixes) != 1: + return None + + # NOTE: All "contract tests" must match `test*.ext`, + # where `.ext` is supported by a registered compiler + elif not file_path.name.startswith("test"): + return None + + from .contracts.collector import ContractTestCollector + + return ContractTestCollector.from_parent(parent, path=file_path) diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py new file mode 100644 index 0000000000..5265e3edd3 --- /dev/null +++ b/tests/integration/conftest.py @@ -0,0 +1,13 @@ +# NOTE: Fixtures for use with both Solidity and Vyper "Contract Tests" feature +import pytest + + +@pytest.fixture(scope="session") +def openzeppelin(project): + dep = project.dependencies["openzeppelin"] + return dep[max(dep)] + + +@pytest.fixture() +def token(openzeppelin, accounts): + return openzeppelin.ERC20Mock.deploy(sender=accounts[-1]) diff --git a/tests/integration/soliditytests/test_basic.sol b/tests/integration/soliditytests/test_basic.sol new file mode 100644 index 0000000000..ee5bd711b4 --- /dev/null +++ b/tests/integration/soliditytests/test_basic.sol @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: UNLICENSED +pragma solidity ^0.8; + +contract BasicTest { + function test_it_works() external { + require(1 + 1 == 2, "We can do tests in solidity!"); + } + + function test_using_fixtures(address[] calldata accounts, address executor) external { + for (uint256 idx = 0; idx < accounts.length; idx++) { + require(accounts[idx].balance >= 10 ** 18); + } + + require(executor == msg.sender); + + bool executor_in_accounts = false; + for (uint256 idx = 0; idx < accounts.length; idx++) { + if (executor == accounts[idx]) { + executor_in_accounts = true; + break; + } + } + require(executor_in_accounts); + } +} diff --git a/tests/integration/soliditytests/test_check_modifiers.sol b/tests/integration/soliditytests/test_check_modifiers.sol new file mode 100644 index 0000000000..1574083b4a --- /dev/null +++ b/tests/integration/soliditytests/test_check_modifiers.sol @@ -0,0 +1,27 @@ +// SPDX-License-Identifier: UNLICENSED +pragma solidity ^0.8; + +import {IERC20} from "openzeppelin/contracts/token/ERC20/IERC20.sol"; + +contract CheckerTest { + /// @custom:ape-check-reverts "This error gets raised" + function test_reverts_with() external { + revert("This error gets raised"); + } + + /// @custom:ape-check-emits + /// - token.Approval(owner=self, spender=executor, value=100_000) + /// - token.Approval(spender=executor, value=10_000) + /// - token.Approval(owner=self, spender=executor) + /// - token.Approval(owner=self, value=100) + /// - token.Approval(value=10) + /// - token.Approval() + function test_emits(IERC20 token, address executor) external { + token.approve(executor, 100_000); + token.approve(executor, 10_000); + token.approve(executor, 1_000); + token.approve(executor, 100); + token.approve(executor, 10); + token.approve(executor, 1); + } +} diff --git a/tests/integration/soliditytests/test_fuzzing.sol b/tests/integration/soliditytests/test_fuzzing.sol new file mode 100644 index 0000000000..4f0f070cab --- /dev/null +++ b/tests/integration/soliditytests/test_fuzzing.sol @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: UNLICENSED +pragma solidity ^0.8; + +import {IERC20} from "openzeppelin/contracts/token/ERC20/IERC20.sol"; + +contract FuzzTest { + /// @custom:ape-fuzzer-max-examples 200 + function test_with_fuzzing(uint256 a) external { + require(a != 29678634502528050652056023465820843, "Found a rare bug!"); + } + + /// @custom:ape-fuzzer-deadline 1000 + function test_token_approvals(IERC20 token, uint256 amount) external { + require(token.approve(msg.sender, amount)); + require(token.allowance(address(this), msg.sender) == amount); + } +} diff --git a/tests/integration/soliditytests/test_mark_modifiers.sol b/tests/integration/soliditytests/test_mark_modifiers.sol new file mode 100644 index 0000000000..d15627fa6a --- /dev/null +++ b/tests/integration/soliditytests/test_mark_modifiers.sol @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: UNLICENSED +pragma solidity ^0.8; + +contract MarkerTest { + /// @custom:ape-mark-xfail "Should not execute" + function test_xfail() external { + revert("Fails for any reason"); + } + + /// @custom:ape-mark-parametrize i + /// - 1 + /// - 2 + /// - 3 + function test_parametrizing(uint256 i) external { + require(i > 0); + } + + /// @custom:ape-mark-parametrize a,b + /// - (0x1, 1) + /// - (0x2, 2) + /// - (0x3, 3) + function test_parametrizing_multiple_args(address a, uint256 b) external { + require(uint256(uint160(a)) == b); + } +} diff --git a/tests/integration/soliditytests/test_stateful.sol b/tests/integration/soliditytests/test_stateful.sol new file mode 100644 index 0000000000..806188a678 --- /dev/null +++ b/tests/integration/soliditytests/test_stateful.sol @@ -0,0 +1,53 @@ +// SPDX-License-Identifier: UNLICENSED +pragma solidity ^0.8; + +/// @custom:ape-fuzzer-max-examples 100 +/// @custom:ape-stateful-step-count 50 +/// @custom:ape-stateful-bundles a b c +contract StatefulTest { + uint256 public secret; + + function setUp() external { + secret = 703895692105206524502680346056234; + } + + /// @custom:ape-stateful-targets a + function initialize_bundleA() external returns (uint256[10] memory) { + // NOTE: Just using static array to return a literal + // (as Solidity automatically casts it) + return [ + uint256(1), + uint256(2), + uint256(3), + uint256(5), + uint256(7), + uint256(11), + uint256(13), + uint256(17), + uint256(19), + uint256(23) + ]; + } + + /// @custom:ape-stateful-precondition this.secret() + a + b < 2 ** 256 + /// @custom:ape-stateful-targets b + function rule_add(uint256 a) external returns (uint256) { + // NOTE: Due to precondition, will **never** fail + secret += a; + + return a % 100; + } + + /// @custom:ape-stateful-consumes b + function rule_subtract(uint256[] calldata a, uint256 b) external { + // NOTE: This will likely fail after a few calls + + for (uint256 idx = 0; idx < a.length; idx++) { + secret -= a[idx] % b; + } + } + + function invariant_secret_not_found() external view { + require(secret != 2378945823475283674509246524589); + } +} diff --git a/tests/integration/soliditytests/test_will_not_execute.t.sol b/tests/integration/soliditytests/test_will_not_execute.t.sol new file mode 100644 index 0000000000..52040990c7 --- /dev/null +++ b/tests/integration/soliditytests/test_will_not_execute.t.sol @@ -0,0 +1,7 @@ +// NOTE: This is only here to show that we will not register "foundry-style" tests as contract tests + +contract Test { + function test_fails_if_registered() external { + require(false, "Something wrong with pytest plugin for contract tests in `src/ape/pytest/plugin.py`."); + } +} diff --git a/tests/integration/soliditytests/test_with_setup.sol b/tests/integration/soliditytests/test_with_setup.sol new file mode 100644 index 0000000000..3cbc416c29 --- /dev/null +++ b/tests/integration/soliditytests/test_with_setup.sol @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: UNLICENSED +pragma solidity ^0.8; + +contract SetupTest { + uint256 store; + + function setUp() external { + store++; + } + + function test_setup_works() external { + require(store == 1); + } + + function test_setup_works_2nd_time() external { + require(store == 1); + } +} diff --git a/tests/integration/vypertests/test_basic.vy b/tests/integration/vypertests/test_basic.vy new file mode 100644 index 0000000000..b5e9175b6b --- /dev/null +++ b/tests/integration/vypertests/test_basic.vy @@ -0,0 +1,32 @@ +# pragma version ~=0.4.3 + + +@external +def test_it_works(): + assert 1 + 1 == 2, "We can do tests in vyper!" + + +@external +def test_using_fixtures(accounts: DynArray[address, 10], executor: address): + """ + @notice + Test cases can use args to access Python fixtures from your Ape test suite. + Ape looks up the fixture by arg name and then provides that to call the method. + + @dev + The fixtures MUST be valid ABI types, or convertible using Ape's conversion system. + + Valid Ape types include: + - `AccountAPI` types (converts to `address`) + - `ContractInstance` types (converts to `address` or interface types) + - strings that Ape's conversion system supports + e.g. `"vitalik.eth"`, `"WETH"`, `"500 USDC"`, etc. + """ + # NOTE: `accounts` is actually an Ape fixture! + for a: address in accounts: + assert a.balance >= 10 ** 18 + + # NOTE: the `executor` fixture is actually the caller of the test + assert executor == msg.sender + # NOTE: the `executor` fixture is in the `accounts` fixture + assert executor in accounts diff --git a/tests/integration/vypertests/test_check_modifiers.vy b/tests/integration/vypertests/test_check_modifiers.vy new file mode 100644 index 0000000000..6ed9fd10ad --- /dev/null +++ b/tests/integration/vypertests/test_check_modifiers.vy @@ -0,0 +1,26 @@ +# pragma version ~=0.4.3 +from ethereum.ercs import IERC20 + +@external +def test_it_raises(): + """ @custom:ape-check-reverts "It works!" """ + assert False, "It works!" + + +@external +def test_emits(token: IERC20, executor: address): + """ + @custom:ape-check-emits + - token.Approval(owner=self, spender=executor, value=100_000) + - token.Approval(spender=executor, value=10_000) + - token.Approval(owner=self, spender=executor) + - token.Approval(owner=self, value=100) + - token.Approval(value=10) + - token.Approval() + """ + extcall token.approve(executor, 100_000) + extcall token.approve(executor, 10_000) + extcall token.approve(executor, 1_000) + extcall token.approve(executor, 100) + extcall token.approve(executor, 10) + extcall token.approve(executor, 1) diff --git a/tests/integration/vypertests/test_fuzzing.vy b/tests/integration/vypertests/test_fuzzing.vy new file mode 100644 index 0000000000..75b8bade68 --- /dev/null +++ b/tests/integration/vypertests/test_fuzzing.vy @@ -0,0 +1,13 @@ +from ethereum.ercs import IERC20 + +@external +def test_with_fuzzing(a: uint256): + """@custom:ape-fuzzer-max-examples 200""" + assert a != 29678634502528050652056023465820843, "Found a rare bug!" + + +@external +def test_token_approvals(token: IERC20, amount: uint256): + """@custom:ape-fuzzer-deadline 1000""" + assert extcall token.approve(msg.sender, amount) + assert staticcall token.allowance(self, msg.sender) == amount diff --git a/tests/integration/vypertests/test_mark_modifiers.vy b/tests/integration/vypertests/test_mark_modifiers.vy new file mode 100644 index 0000000000..1d9932557e --- /dev/null +++ b/tests/integration/vypertests/test_mark_modifiers.vy @@ -0,0 +1,31 @@ +# pragma version ~=0.4.3 + + +@external +def test_xfail(): + """ + @custom:ape-mark-xfail "Should not execute" + """ + raise "Fails for any reason" + + +@external +def test_parametrizing(i: uint256): + """ + @custom:ape-mark-parametrize i + - 1 + - 2 + - 3 + """ + assert i > 0 + + +@external +def test_parametrizing_multiple_args(a: address, b: uint256): + """ + @custom:ape-mark-parametrize a,b + - (0x1, 1) + - (0x2, 2) + - (0x3, 3) + """ + assert convert(a, uint256) == b diff --git a/tests/integration/vypertests/test_stateful.vy b/tests/integration/vypertests/test_stateful.vy new file mode 100644 index 0000000000..698dd67922 --- /dev/null +++ b/tests/integration/vypertests/test_stateful.vy @@ -0,0 +1,90 @@ +""" +@custom:ape-fuzzer-max-examples 100 +@custom:ape-stateful-step-count 50 +@custom:ape-stateful-bundles a b c +""" + +secret: public(uint256) + + +@external +def setUp(): + self.secret = 703895692105206524502680346056234 + + +@external +def initialize_bundleA() -> DynArray[uint256, 10]: + """ + @notice + Add some initial values to a bundle. "Initializers" are called exactly once at the + beginning of a test (before any `rule`s are called), but could be called in any order. + They are mostly used to initialize "Bundles" with values for the rest of the test. + + @dev + Same as `@initializes` in Hypothesis, allowing to set up initial test state (incl Bundles). + The return value is injected into the bundle specified by `@custom:test:stateful:targets`. + A single return (1 value) or array return (multiple values) are supported for conveinence. + + @custom:ape-stateful-targets a + """ + return [1, 2, 3, 5, 7, 11, 13, 17, 19, 23] + + +@external +def rule_add(a: uint256) -> uint256: + """ + @notice + A rule is an action that MAY be called by the test harness as one step in the test. + + Rules can also return Bundles values, which add more choices to the associated bundle. + + @dev + A rule is selected at random, and follows the same rules as normal tests with regard to arguments. + + If you wish to avoid calling a rule except under a particular scenario, add a precondition. + + @custom:ape-stateful-precondition self.secret() + a + b < 2**256 + @custom:ape-stateful-targets b + """ + # NOTE: Due to precondition, will **never** fail + self.secret += a + + return a % 100 + + +@external +def rule_subtract(a: DynArray[uint256, 10], b: uint256): + """ + @notice + If a failure occurs when executing a rule, that will automatically raise a test failure. + + This may indicate a legitimate bug in what you are testing, or a design flaw in your test. + + @dev + Each argument that has a name matching a bundle "pulls" values from the associated bundle. + If `@custom:test:stateful:consumes` is present, then that value will instead be "consumed" + by the rule, and therefore removed from the associated bundle (e.g. no longer available) + + To pull multiple values from a bundle, use an array (selection size is chosen at random). + + @custom:ape-stateful-consumes b + """ + # NOTE: This will likely fail after a few calls + + for val: uint256 in a: + self.secret -= val % b + + + +@view +@external +def invariant_secret_not_found(): + """ + @notice + An invariant is called after every rule invocation, to check consistency of internal state. + If it fails, it will automatically raise a test failure, likely indicating a legitimate bug. + + @dev + An invariant **MUST** be `view`/`pure` mutability or it will be ignored. + """ + assert self.secret != 2378945823475283674509246524589 diff --git a/tests/integration/vypertests/test_will_not_execute.t.vy b/tests/integration/vypertests/test_will_not_execute.t.vy new file mode 100644 index 0000000000..3d5d90d985 --- /dev/null +++ b/tests/integration/vypertests/test_will_not_execute.t.vy @@ -0,0 +1,5 @@ +# NOTE: This is only here to show that we will not register "foundry-style" tests as contract tests + +@external +def test_fails_if_registered(): + raise "Something wrong with pytest plugin for contract tests in `src/ape/pytest/plugin.py`." diff --git a/tests/integration/vypertests/test_with_setup.vy b/tests/integration/vypertests/test_with_setup.vy new file mode 100644 index 0000000000..cc238dafc0 --- /dev/null +++ b/tests/integration/vypertests/test_with_setup.vy @@ -0,0 +1,18 @@ +# pragma version ~=0.4.3 + +store: uint256 + + +@external +def setUp(): + self.store += 1 + + +@external +def test_setup_works(): + assert self.store == 1 + + +@external +def test_setup_works_2nd_time(): + assert self.store == 1