Implement base tests for frooky host

[bernhste]

Addresses #7

Testing was previously tracked in #52. This PR now only implements CLI tests for the Python app using pytest.

This PR is used to test the Python host. For example, the argument parsing, JSON/YAML parsing, or parsing of the results sent back from the agent. They can also be end-to-end, but then we need to have the proper environment (#25).

At the moment, I implemented only two PoC tests. The goal of this PR is to establish a good foundation for Python/CLI testing.

[Diolor]

I installed it with pip install -e ".[dev]". I run it locally and it's great!

• Let's also remember to update our development guide on how to run these tests

[bernhste]

Lots of trial and error, but now Android integration tests and two unit tests run locally:

(The iOS tests are stubs.)

And more importantly, they also work remotely.

There is still room for optimization, but the tests run well. Also made a small graph how the test runs:

Main Thread
|
+-- Start frooky_process (background) -----> (writes to output.json)
|
+-- run_maestro() (blocks until complete)
|
+-- terminate frooky_process
|
+-- assertions

There are no timeouts, etc. The output is only parsed after maestro finishes with the UI interaction.

@cpholguera I took your Android runtime as a template and implemented integration tests in pytest. Please look at this test so we can decide how to continue.

My testing recommendation would be:

• Regular unit tests for frooky agents (TypeScript Implement base tests for frooky agents  #58) and host (Python pytest). This means focus on testing errors, input parsing, or exception handling.
• Integration tests with the frooky host and a target app using pytest. Here we should focus on good cases. Because we should avoid testing errors or exceptions we already covered with unit tests.

If that's ok, we should document how to write good unit/integration tests.

[bernhste]

Update: I generalized the pattern matcher.

Basically, you can now define a subset of what objects must be in the output.json. This makes integration tests easy. An example:

I want to test if the overload works properly. My hook.json:

{
  "category": "STORAGE",
  "hooks": [
    {
      "class": "androidx.security.crypto.EncryptedSharedPreferences",
      "method": "create",
      "overloads": [
        {
          "args": [
            "android.content.Context",
            "java.lang.String",
            "androidx.security.crypto.MasterKey",
            "androidx.security.crypto.EncryptedSharedPreferences$PrefKeyEncryptionScheme",
            "androidx.security.crypto.EncryptedSharedPreferences$PrefValueEncryptionScheme"
          ]
        }
      ]
    }
  ]
}

To test this, I expect the output.json to have an object that contains this SUBSET:

expected_patterns = [
    {
        "class": "androidx.security.crypto.EncryptedSharedPreferences",
        "method": "create",
        "inputParameters": [
                {
                    "declaredType": "android.content.Context"
                },
                {
                    "declaredType": "java.lang.String"
                },
                {
                    "declaredType": "androidx.security.crypto.MasterKey"
                },
                {
                    "declaredType": "androidx.security.crypto.EncryptedSharedPreferences$PrefKeyEncryptionScheme"
                },
                {
                    "declaredType": "androidx.security.crypto.EncryptedSharedPreferences$PrefValueEncryptionScheme"
                }
            ],
    }
]

This will match the position, key/value, and length of lists recursively.

If there are more elements in the target, they are ignored since we care that at least THIS pattern is in the `output.json as expected.

So now it is easy to define integration tests for all kinds of use cases.

[bernhste]

Finally, we have working unit and integration tests for the frooky host!

We can test everything locally from the root folder with:

• pytest: Test everything
• pytest tests/unit: Unit tests
• pytest tests/integration/ios: Integration tests iOS
• pytest tests/integration/android: Integration tests Android

This should look something like that:

The integration tests should have all information within the test method:

• Hook declaration
• Expected output in the output.json

This is an example:

def test_hook_native(self, run_frooky, number_of_matched_events, output_file_path):
    """Test hooking a single Java method in a real process."""

    hook = {
        "category": "STORAGE",
        "hooks": [
            {
                "native": True,
                "symbol": "open",
                "maxFrames": 5,
                "filterEventsByStacktrace": "MASTestApp",
                "args": [
                    {
                        "type": "string",
                        "name": "path",
                        "filter": ["Containers/Data/Application/"]
                    }
                ]
            }
        ]
    }

    run_frooky(hook)

    expected_event = {
            "type": "native-hook",
            "symbol": "open",
            "inputParameters": [
                {
                    "type": "string",
                    "name": "path"
                }
            ],
        }

    assert output_file_path.exists(), "output.json was not created"
    assert number_of_matched_events(expected_event) == 4, "Not the amount of expected matched events found."

number_of_matched_events(expected_event) counts the number of events which match expected_event (which can a subset only).

There are still a few open points:

• Documentation
• Writing tests for existing code
• Writing a dedicated, stable target app
• Improve performance for the build infrastructure (e.g. better caching of simulators and apps, decoupling it from the test etc.)
• Upload a nice output from the test results like https://github.com/cpholguera/frooky/actions/runs/21754977797

@cpholguera You mentioned, that you want to expand the test apps. I think this is very helpful, because I had a few issues with confusing API and expected results with the DEMO-App. So it would be very helpful if we have a stable target where we can write test against without ambiguities or complicated API calls. Also, we should clean up the App Compilation / Build / Simulator staging do speed up the verification.

But let's do that in a second PR.