Add ability to save the result and other context when a check fails.

[steverivers]

My team currently uses k6 for performance and functional testing. One of the initial problems when we switched to k6 from our old tools was including the http request / response of failing tests into reports. We solved that by creating a fairly simple results store that we POST the test context {test_name, id, Response object etc.} to. As we prefer the chaijs fluent BDD style I created a helper that extends the assertions to send user supplied context data to our results store for reporting.

However, I'm assuming that this project is where the future of functional testing support is going to be developed. Is this something that is being considered for future work?

This is what I have come up with as a very simple proof of concept, inserting the following into expextNonRetrying.ts

  const result = checkFn();
  // If isNegated is true, we want to invert the result
  const finalResult = isNegated ? !result : result;
  if (!finalResult) {

    if (matcherSpecific?.request && matcherSpecific?.response) {
      //saveResult(matcherSpecific.request, matcherSpecific.response);
      console.log(
        "HTTP Request:",
        JSON.stringify(matcherSpecific.request, null, 2),
      );
      console.log(
        "HTTP Response:",
        JSON.stringify(matcherSpecific.response, null, 2),

      );
    } else {
      console.log("No HTTP request/response available");
    }
  }
  usedAssert(
    finalResult,
    MatcherErrorRendererRegistry.getRenderer(matcherName).render(
      info,
      MatcherErrorRendererRegistry.getConfig(),
    ),
    isSoft,
  );

[oleiade]

Hi @steverivers 👋🏻

Thanks a lot for your feedback. This library's content is indeed meant to become part of k6 itself down the road. Both to support browser testing, and synthetic checking with k6 better, and in a more canonical way.

I'm not entirely certain I have enough context myself, or understand thoroughly the need you have. Could you please provide an example script using a fictional version of the library that would perform what you aim for?

My current level of understanding is that you're looking to kind of have hooks into assertions lifecycle so that you can automatically plug custom code, to, for instance log specific information, on certain conditions?

[steverivers]

@oleiade - Thanks for the reply, it seems I was not as clear as I thought I'd been :( . I will try and put together a sample this week to better explain what and why, and will drop it here as soon as I'm done.

[steverivers]

@oleiade
I've created a sample project - https://github.com/steverivers/functional_logging_example hope it and the below description helps explain what I was talking about.

Coincidentally, based on a couple of meetings today, I've had a rethink and I'm not sure this is the right place for what i want. Instead of shoehorning this feature into the testing library. It might be better if the data that I'm looking for (request, response etc.) is emitted as OpenTelemetry logging alongside the existing metrics.

This seems to have multiple benefits

• Keeps the testing library focus on testing not logging.
• Uses industry standards and tools
• Allows for correlation between test and application

Thanks for your time and sorry for wasting it
Steve

Overview of original ask for posterity

We use k6 to do a range of functional regression testing, data validation testing etc. When an assertion fails we want to know the details of what was sent and recieved. The previous tool we used, ReadyApi, allowed access to the HTTP request and response on failure and having this info in the reports reduces the friction during the troubleshooting / fixing.

Initially we were using the built-in check functioanlity and if the check failed we'd send the response object along with some other context data to a simple express.js server i worte to save / retrieve the results.

    let response = make_request_to_api(HOST, body, params);

    let statusCodeCheck = check(response, {
      'response code is 200': (response) => response.status == 200,
    });
    if (!statusCodeCheck) {
      saveResponse(response)
    }

However, the team much prefer to use the BDD style from Chai and this does not support the flow above.

Therefore, I created a chai helper to extend both standard and chainable assertions to allow the saving of the response and other contect data as before.

The save feature is only used during our functional test runs to limit the stress on the results server. To enable logging the options object needs some tags added as below

export const options = {
  tags: {
    type: "functional",
    name: "sample test to demonstrate our custom exception logging",
    testId: uuidv4(),
  },
};

Example

Hopefully the example is fairly self-explanatory.

The SampleTest makes a couple of requests to an api.

The first test passes and the second fails. The test report shows the http request and response on the "Failed Check Details" tab

[oleiade]

I see, thanks a bunch for the detailled explanation, that was super useful, and valuable 🙇🏻

My take is that we're pretty much aligned with your statement that it's probably not the role of the testing library to handle what sounds like a logging/otel job indeed.

For both UX/DX and strategic reasons, this library is trying to be as compatible as possible with Playwright's assertions. Their test library does offer the ability to create custom assertions, and I do wonder if we were to add support for it, could it actually be one viable solution to your need?

What are your thoughts?

[steverivers]

Thanks, that makes a lot of sense, we also use Playwright quite heavily as well, so supporting the custom assertions model would be great.

From a "doing our job" perspective, my current solution meets our needs at this time and should be straightforward to migrate once this library is ready.

I'm going to try and keep active with this effort, but I'm still waiting for clarification from higher ups about contributing back to OSS on company time.