stateful fuzz with foundry doesn't seem to call all functions in the contract

[aadarshaddy]

In this video at 6:12 Patrick says foundry is smart enough to fuzz all public fuzzable functions. so this is my contract:

// SPDX-License-Identifier: MIT
pragma solidity 0.8.20;

// INVARIANT: doMoreMathAgain should never return 0
contract StatefulFuzzCatches {
    uint256 public myValue = 1;
    uint256 public storedValue = 100;

    /*
     * @dev Should never return 0
     */
    function doMoreMathAgain(uint128 myNumber) public returns (uint256) {
        uint256 response = (uint256(myNumber) / 1) + myValue;
        storedValue = response;
        return response;
    }

    function changeValue(uint256 newValue) public {
        myValue = newValue;
    }
}

this is my test file:

// SPDX-License-Identifier: MIT
pragma solidity 0.8.20;

import {Test} from "forge-std/Test.sol";
import {StdInvariant} from "forge-std/StdInvariant.sol";
import {StatefulFuzzCatches} from "../../src/invariant-break/StatefulFuzzCatches.sol";

contract StatefulFuzzCatchesTest is StdInvariant, Test {
    StatefulFuzzCatches sfc;

    function setUp() public {
        sfc = new StatefulFuzzCatches();
        targetContract(address(sfc));
    }

    // // Stateful fuzz
    // // also can use the prefix `invariant_`
    // // Uncomment this to see it catch the issue!
    function invariant_testMathDoesntReturnZero() public view {
        assert(sfc.storedValue() != 0);
    }
}

foundry.toml file has the following changes:

[fuzz]
runs = 1000000
seed = '0x2'

[invariant]
runs = 64
depth = 32
fail_on_revert = false

It is exactly like in the https://github.com/Cyfrin/sc-exploits-minimized/blob/main/test/invariant-break/StatefulFuzzCatchesTest.t.sol file, When I run invariant_testMathDoesntReturnZero it always passes and does't fail at all.

$ forge test --mt invariant_testMathDoesntReturnZero -vvv   
[⠊] Compiling...
[⠒] Compiling 1 files with Solc 0.8.20
[⠢] Solc 0.8.20 finished in 1.04s
Compiler run successful!

Ran 1 test for test/invariant-break/StatefulFuzzCatchesTest.t.sol:StatefulFuzzCatchesTest
[PASS] invariant_testMathDoesntReturnZero() (runs: 64, calls: 2048, reverts: 39)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 95.89ms (89.60ms CPU time)

Ran 1 test suite in 282.77ms (95.89ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)

Am I doing anything wrong here or did Foundry have an update where it does not fuzz all public functions anymore after the course came out?

[EngrPips]

The fail_on_revert in this GitHub https://github.com/Cyfrin/sc-exploits-minimized/blob/main/test/invariant-break/StatefulFuzzCatchesTest.t.sol you shared is set to true, and yours is set to false. Any reason for that?

[EngrPips]

If he has it set to true in his foundry.toml does that mean he ran into the overflow error in his test too?

[Kidhapelmzy]

yes he ran into overflow error, set it to false and got assertion violated

[EngrPips]

Use seed = 0x1 and then run forge clean, then retry the test, let's see if it passes still.

[aadarshaddy]

@EngrPips changed seed = 0x1 , forge clean . still no luck. One thing I noticed is even though runs = 100000 following is the forge test response:

% forge test --mt invariant_testMathDoesntReturnZero -vvv
[⠊] Compiling...
[⠒] Compiling 69 files with Solc 0.8.20
[⠑] Solc 0.8.20 finished in 1.56s
Compiler run successful with warnings:
Warning (5159): "selfdestruct" has been deprecated. The underlying opcode will eventually undergo breaking changes, and its use is not recommended.
  --> src/mishandling-of-eth/SelfDestructMe.sol:45:9:
   |
45 |         selfdestruct(payable(address(target)));
   |         ^^^^^^^^^^^^


Ran 1 test for test/invariant-break/StatefulFuzzCatchesTest.t.sol:StatefulFuzzCatchesTest
[PASS] invariant_testMathDoesntReturnZero() (runs: 64, calls: 2048, reverts: 13)
Suite result: ok. 1 passed; 0 failed; 0 skipped; finished in 108.22ms (106.97ms CPU time)

Ran 1 test suite in 294.99ms (108.22ms CPU time): 1 tests passed, 0 failed, 0 skipped (1 total tests)

It ran only 64 times?
AI recommends workaround to add a handler contract to call other functions, but that defeats the point of "automated fuzzing" and becomes manual fuzzing then right?

[EngrPips]

Using a Handler contract does not defeat fuzzing, but this particular contract is too simple to use an Handler contract for. You can try changing the seed to different values, and I believe you will later find the seed that will make the test fail.

[Kidhapelmzy]

i am facing the same problem. The test is supposed to fail (Patrick's example did) with assertion violation as the reason, but it passed.

[EngrPips]

I responded to your discussion. You can also share the link to the lesson as well.

[aadarshaddy]

I was changing the runs under [fuzz] (stateless) and not under [invariant] (stateful). and hence it did not work for me.
@Kidhapelmzy @EngrPips

[fuzz]
runs = 1000000
seed = '0x2'

[invariant]
runs = 10000 ##this worked
depth = 32
fail_on_revert = false

[EngrPips]

Awesome, I am glad you figured it out. Keep crushing it.