Skip to content
This repository was archived by the owner on Aug 19, 2025. It is now read-only.

RFC #1 - test entry point #20

Open
lawnsea wants to merge 1 commit into
base: master
Choose a base branch
from
Open

RFC #1 - test entry point #20

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions doc/rfc/1-test-entry-point.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Summary

## Problem

We want ingredients and pantries to be easy to test. Specifically, it should be
easy to write both unit and functional tests, easy to run those tests, and easy
to obtain coverage data.

This RFC proposes the ingredient interface to support these goals.

## Solution Overview

Ingredients may define a test entry point (`test.js`) that runs the ingredient's
unit and functional tests and reports [TAP](https://testanything.org/) results.
The test harness can be configured to ensure a Roux server is running before
executing the test entry point. If it does so, it will provide the URL for that
server via an environment variable.
The test harness will be able to execute the test entry points of multiple
ingredients and report the aggregate results.

# API

## Test entry point

An ingredient MAY define a test entry point by including a file named `test.js`
in the ingredient root. When invoked, the script must print a [Test Anything
Protocol (TAP)](https://testanything.org/) test stream to `stdout` before
exiting. The script MAY run one or more tests when invoked. The result of any
tests the script runs MUST be printed to the TAP stream. If any test fails, the
script MUST exit with a non-zero exit code. Otherwise, it MUST exit with an exit
code of zero.

The test entry point MAY receive configuration via its `process.argv` and
Copy link
Contributor

@lzilioli lzilioli Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs to be fleshed out a bit further. If I am invoking tests for multiple ingredients from the command line, will all arguments passed to process.argv be passed to all the ingredients? What if two ingredients allow for the same flags but interpret them to mean different things?

Random thought that is probably a bad idea: Perhaps using a tool like commander to roll up, parse, and generate a help message for the options allowed by each ingredient being run for this invocation of the test runner would be in order? That would give us a chance to prevent two ingredients from expecting the same flag with a different meaning.

`process.env` properties.

The test entry point MUST NOT assume a Roux server is running. If a Roux server
Copy link
Contributor

@lzilioli lzilioli Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its not clear to me what the test would do differently in a case where the server is running vs. where it is not? I'm still on board with the server being running being optional, however if a test.js bypasses an entire subset of its test suite because the server is not running, I would not be particularly comfortable with the test.js file exiting 0 in this case, as it is not fully confident that the ingredient passes all of its test.

is running, the test entry point WILL receive the root URL for its ingredient as
an environment variable named `ROUX_INGREDIENT_URL`.

A test entry point MUST NOT assume that it is the only test entry point
executing. The test harness MAY execute multiple entry points concurrently, so
access to any shared resources MUST be coordinated via other means.