Initial commit of compliance test repo

This has extracted from the boto/jmespath repo and
pulled into its own repo.

Author: jamesls

README.rst

@@ -0,0 +1,103 @@
+JMESPath Compliance Tests
+=========================
+
+This repo contains a suite of JMESPath compliance tests.  JMESPath
+implementations can use these tests in order to verify their implementation
+adheres to the JMESPath spec.
+
+Test Organization
+=================
+
+The ``test/`` directory contains JSON files containing the JMESPath testcase.
+Each JSON file represents a JMESPath feature.  Each JSON file is
+a JSON list containing one or more tests suites::
+
+    [
+      <test suite 1>,
+      <test suite 2>,
+    ]
+
+Each test suite is a JSON object that has the following keys:
+
+* ``given`` - The input data from which the JMESPath expression is evaluated.
+* ``cases`` - A list of test cases.
+* ``comment`` - An optional field containing a description of the test suite.
+
+Each JMESPath test case can have the following keys:
+
+* ``expression`` - The JMESPath expression being tested.
+* ``result`` - The expected result from evaluating the JMESPath expression
+  against the ``given`` input.
+* ``error`` - The type of error that should be raised as a result of evaluating
+  the JMESPath expression.
+* ``comment`` - An optional comment containing a description of the specific
+  test case.
+
+For each test case, either ``result`` or ``error`` must be specified.  Both
+keys cannot be present in a single test case.
+
+The error type (if the ``error`` key is present) indicates the type of error
+that an implementation should raise, but it does not indicate **when** this
+error should be raised.  For example, a value of ``"error": "syntax"`` does not
+require that the syntax error be raised when the expression is compiled.  If an
+implementation does not have a separate compilation step this won't even be
+possible.  Similar for type errors, implementations are free to check for type
+errors during compilation or at run time (when the parsed expression is
+evaluated).  As long as an implementation can detect that this error occured at
+any point during the evaluation of a JMESPath expression, this is considered
+sufficient.
+
+Below are a few examples::
+
+    [{
+        "given":
+            {"foo": {"bar": {"baz": "correct"}}},
+         "cases": [
+             {
+                "expression": "foo",
+                "result": {"bar": {"baz": "correct"}}
+             },
+             {
+               "expression": "foo.1",
+               "error": "syntax"
+             },
+        ]
+    }]
+
+This above JSON document specifies 1 test suite that contains 2 test cases.
+The two test cases are:
+
+* Given the input ``{"foo": {"bar": {"baz": "correct"}}}``, the expression
+  ``foo`` should have a result of ``{"bar": {"baz": "correct"}}``.
+* Given the input ``{"foo": {"bar": {"baz": "correct"}}}``, the expression
+  ``foo.1`` should generate a syntax error.
+
+
+Utility Tools
+=============
+
+Most languages have test frameworks that are capable of reading the JSON test
+descriptions and generating testcases.  However, a ``jp-compliance`` tool is
+provided to help with any implementation that does not have an available test
+framework to generate test cases.  The ``jp-compliance`` tool takes the name of
+a jmespath executable and will evaluate all the compliance tests using this
+provided executable.  This way all that's needed to verify your JMESPath
+implementation is for you to write a basic exectuable.  This executable must
+have the following interface:
+
+* Accept the input JSON data on stdin.
+* Accept the jmespath expression as an argument.
+* Print the jmespath result as JSON on stdout.
+* If an error occurred, it must write the error name to sys.stderr.  The error
+  name must be the first thing written to stderr, but it is free to write
+  information after the error name.  For example "syntax: Expression did not
+  compile 'foo.bar.'" would be a valid message to indicate that a syntax error
+  occurred because it starts with "syntax".
+
+.. note::
+
+  This will be substantially slower than using a test framework.  Using
+  ``jp-compliance`` each test case is evaluated by executing a new process.
+
+You can run the ``bin/jp-compliance --help`` for more information and for
+examples on how to use this tool.

tests/basic.json

@@ -0,0 +1,92 @@
+[{
+    "given":
+        {"foo": {"bar": {"baz": "correct"}}},
+     "cases": [
+         {
+            "expression": "foo",
+            "result": {"bar": {"baz": "correct"}}
+         },
+         {
+            "expression": "foo.bar",
+            "result": {"baz": "correct"}
+         },
+         {
+            "expression": "foo.bar.baz",
+            "result": "correct"
+         },
+         {
+            "expression": "foo.bar.baz.bad",
+            "result": null
+         },
+         {
+            "expression": "foo.bar.bad",
+            "result": null
+         },
+         {
+            "expression": "foo.bad",
+            "result": null
+         },
+         {
+            "expression": "bad",
+            "result": null
+         },
+         {
+            "expression": "bad.morebad.morebad",
+            "result": null
+         }
+     ]
+},
+{
+    "given":
+        {"foo": {"bar": ["one", "two", "three"]}},
+    "cases": [
+         {
+            "expression": "foo",
+            "result": {"bar": ["one", "two", "three"]}
+         },
+         {
+            "expression": "foo.bar",
+            "result": ["one", "two", "three"]
+         }
+    ]
+},
+{
+    "given": ["one", "two", "three"],
+    "cases": [
+        {
+            "expression": "one",
+            "result": null
+        },
+        {
+            "expression": "two",
+            "result": null
+        },
+        {
+            "expression": "three",
+            "result": null
+        },
+        {
+            "expression": "one.two",
+            "result": null
+        }
+    ]
+},
+{
+    "given":
+        {"foo": {"1": ["one", "two", "three"], "-1": "bar"}},
+    "cases": [
+         {
+            "expression": "foo.\"1\"",
+            "result": ["one", "two", "three"]
+         },
+         {
+            "expression": "foo.\"1\"[0]",
+            "result": "one"
+         },
+         {
+            "expression": "foo.\"-1\"",
+            "result": "bar"
+         }
+    ]
+}
+]

tests/escape.json

@@ -0,0 +1,46 @@
+[{
+    "given": {
+        "foo.bar": "dot",
+        "foo bar": "space",
+        "foo\nbar": "newline",
+        "foo\"bar": "doublequote",
+        "c:\\\\windows\\path": "windows",
+        "/unix/path": "unix",
+        "\"\"\"": "threequotes",
+        "bar": {"baz": "qux"}
+     },
+     "cases": [
+         {
+            "expression": "\"foo.bar\"",
+            "result": "dot"
+         },
+         {
+            "expression": "\"foo bar\"",
+            "result": "space"
+         },
+         {
+            "expression": "\"foo\\nbar\"",
+            "result": "newline"
+         },
+         {
+            "expression": "\"foo\\\"bar\"",
+            "result": "doublequote"
+         },
+         {
+            "expression": "\"c:\\\\\\\\windows\\\\path\"",
+            "result": "windows"
+         },
+         {
+            "expression": "\"/unix/path\"",
+            "result": "unix"
+         },
+         {
+            "expression": "\"\\\"\\\"\\\"\"",
+            "result": "threequotes"
+         },
+         {
+            "expression": "\"bar\".\"baz\"",
+            "result": "qux"
+         }
+     ]
+}]