chore(cli): consume direct geth blocktest fail fix (ethereum#1502)

* chore(cli): consume direct geth blocktest fail fix.

* chore(docs): add changelog.

Author: spencer-tb

docs/CHANGELOG.md

@@ -26,6 +26,8 @@ Users can expect that all tests currently living in [ethereum/tests](https://git
 
 #### `consume`
 
+- 🐞 Fix consume direct fails for geth blockchain tests ([#1502](https://github.com/ethereum/execution-spec-tests/pull/1502)).
+
 ### 📋 Misc
 
 ### 🧪 Test Cases

src/ethereum_clis/clis/geth.py

@@ -268,6 +268,18 @@ def consume_blockchain_test(
                 f"Unexpected exit code:\n{' '.join(command)}\n\n Error:\n{result.stderr}"
             )
 
+        result_json = json.loads(result.stdout)
+        if not isinstance(result_json, list):
+            raise Exception(f"Unexpected result from evm blocktest: {result_json}")
+
+        if any(not test_result["pass"] for test_result in result_json):
+            exception_text = "Blockchain test failed: \n" + "\n".join(
+                f"{test_result['name']}: " + test_result["error"]
+                for test_result in result_json
+                if not test_result["pass"]
+            )
+            raise Exception(exception_text)
+
     @cache  # noqa
     def consume_state_test_file(
         self,