Unit/integration testing durable functions

[paulschroeder-tomtom]

💡 Feature description
We were using durable functions to create an API (what we call the orchestration API - oAPI). This one processes requests and (HTTP-) calls other (primitive - our terminology) APIs (pAPI). We have created a durable function (DF) app and registered several functions via decorators for the different routes. I.e.

import azure.durable_functions as dfapp = df.DFApp(http_auth_level=func.AuthLevel.ANONYMOUS)

app.register_blueprint(projects_orchestrators)
app.register_blueprint(landing_page_bp)# /projects

@app.route(route="projects")
@app.function_name(name="projects")
@app.durable_client_input(client_name="client")
async def projects(req: func.HttpRequest, context: func.Context, client) -> func.HttpResponse:
    ...
    instance_id = await client.start_new("list_projects_orchestrator", client_input=client_input)
    ...
    retrun client.create_check_status_response(req, instance_id)

@projects_orchestrators.orchestration_trigger(context_name="context")
def list_projects_orchestrator(context: df.DurableOrchestrationContext):
    ...
    # the real magic happens here
    ...

💭 Describe alternatives you've considered
What we now would like to do now, is to (integration) test the whole setup and treat the DF app like a black box. So our test should look like so:

@pytest.mark.asyncio
async def test_projects():
    # mock pAPIs
    httpretty.register_uri(httpretty.GET, 'https://papi1.com/foo', body=json.dumps({'data': 123}))
    httpretty.register_uri(httpretty.GET, 'https://papi2.com/bar', body=json.dumps({'data': 456}))

    request = func.HttpRequest(...) 
    
    # pseudocode code
    from function_app import projects as function_under_test
    result = await function_under_test(req=request, context, client)
    # /pseudocode code 

    assert result == ...

We were already doing quite some tinkering to even get the equivalent to result = await function_under_test(req=request, context, client) working. But everything is quite fragile and is still failing because, as we guess, stuff that is supplied by the runtime is still missing.

We also would really like to avoid any shell command calls to bring up the runtime and prefer to have everything in python to easily mock external dependencies (we are also able to emulate the storage via azurite).

So, our burning questions are:

• is the way we are trying to go a dead end?
• how do we correctly call the functions like project from above?
• how do we correctly mock the inputs like context and client of projects?
• is it even possible to bring up the runtime purely in python/pytest?
• if so, how?

Thank you!

[davidmrdavid]

Hi @paulschroeder-tomtom, thanks for reaching out.

Yeah we've received a bit of feedback about the lack of clarity for testing, so we realize this is an area that needs improvement. I think we can use your project as a driving case to try to improve the experience here.

So to answer your question directly: I am personally not immediately sure that there's a "best practice"/ approved way to mock the DF Client and DF Context today (but there should be!), but I'm pretty certain that it is possible at the expense of it being hacky Are these the main objects you're struggling to mock? If so, I can try to put together the utilities that would allow you to do this and release them soon-ish.

In general, if you can put together a minimal app (emphasis on minimal, so that it's something I can run on my end) that you can share with me and comment in there the specific functionality you're and struggling trying to mock, that would help me identify the missing utilities (if any) and help provide guide the testing strategy.

[brandonwatts]

@davidmrdavid We are also struggling to really unittest our durable functions in v2. The decorators while really nice to work with, are a pain to unravel when testing. In activity functions we are testing them like <function_name>._function._func(<inputs) which isnt a huge deal but the orchestrators are almost impossible to test. Ill give an example. Say you wanted to write a unittest for the following orchestrator:

@bp.orchestration_trigger(context_name="context")
def orchestrator(context: df.DurableOrchestrationContext):
    x: int = context.get_input()
    y = yield context.call_activity(plusone, x)
    return y

@bp.activity_trigger(input_name="number")
def plusone(number: int) -> int:
    return number + 1

It isnt immediately apparent to me how you would do this. We copied over a contextbuilder class similar to how you guys did here but it ends up bringing with it almost the entire DF project. Its brittle and just overly complex. Im not sure what the right answer is here but I would love to have a simple way to test orchestrators and suborchestrators.

[paulschroeder-tomtom]

Hey @davidmrdavid

totally agree with @brandonwatts . I quickly coded up a minimal example:

import logging

import azure.durable_functions as df
import azure.functions as func

ips_orchestrators = df.Blueprint()
app = df.DFApp(http_auth_level=func.AuthLevel.ANONYMOUS)


# /ips
@app.route(route="ips/{version}")
@app.function_name(name="ips")
@app.durable_client_input(client_name="client")
async def ips(req: func.HttpRequest, context: func.Context, client) -> func.HttpResponse:
    """
    Could be more complex, so it needs to be tested. Kept short just for the sake of this example.
    NOTE:
        As @brandonwatts said, the decorators "eat" up the function (I guess durable_client_input is returning
        None instead of the original, other decorators also magle it from a regular Callable to DF) and so
        the ips() does not end up in the scope if this file after importing it.
    """
    # ... do something
    # ... do something with the context
    client_input = dict(
        des_version=req.route_params.get('version')
    )

    instance_id = await client.start_new('list_ips_orchestrator', client_input=client_input)
    # ... do something more
    return client.create_check_status_response(instance_id=instance_id, request=req)


# Construct and send primitive API calls to create project
@ips_orchestrators.orchestration_trigger(context_name="context")
def list_ips_orchestrator(context: df.DurableOrchestrationContext):
    """
    I know about context.task_all() but I wanted to keep it like that, because it reflects more our use case.
    Same here with the decorators
    """
    des_version = context.get_input()['des_version']

    ip_v4 = yield context.call_http(
        method='GET',
        uri='https://ipv4.icanhazip.com',
    )
    ip_v6 = yield context.call_http(
        method='GET',
        uri='https://ipv6.icanhazip.com',
    )

    return dict(
        des_verison=des_version,
        ip_v4=ip_v4['content'],
        ip_v6=ip_v6['content'],
    )


app.register_blueprint(ips_orchestrators)

[paulschroeder-tomtom]

It already 3 months ago that we opened up that issue. Is there any development on that?

[lilyjma]

Hi @paulschroeder-tomtom , @brandonwatts - thank you for using Durable Functions! I'm a PM working on DF and would love to learn about your experience using the product. You can share your feedback in this quick survey to help influence what the team works on next. If you're building intelligent apps, there's also an opportunity to participate in a compensated UX study. Thanks!

[paulschroeder-tomtom]

Hey @lilyjma to share my feedback also here: I am still waiting for member of the DF team to appropriately answer on that request right here as well as several other bugs/issues/question and feature requests.

I am quite annoyed by the behavior of MS. You are always asking for more information, a minimal example or (like here) a survey, without anything in return. No solution, no help. It really feels one sided! Improve on that.

[lilyjma]

HI @paulschroeder-tomtom - thanks for the feedback and apologies for the delayed response. My original intention was to get back to you and Brandon after we wrap up planning, because that way I can tell you whether this is something we'll work on next. (The survey was intended to gather data to help with work prioritization for the upcoming semester.) That said, I do acknowledge that the turnaround time wasn't great, so I understand your frustration here.

The good news is that helping customers easily write Python unit tests is something we'll prioritize in the next semester, which starts in April. I can't give you a definitive date now on when the improved experience will go out, since there's existing current semester work that needs wrap up, but we do see the feedback here and will address this ASAP when in progress work is done. I'll be sure to keep this thread updated.

I'll add @davidmrdavid to chime in with anything he'd like to add.

[cjsimm]

Any movement on this? Also searching for a reliable way to mock and run integration tests on a durable app.