Skip to content

Commit bc38bb9

Browse files
author
MarcoFalke
committed
Merge #17288: Added TestShell class for interactive Python environments.
19139ee Add documentation for test_shell submodule (JamesC) f511236 Add TestShell class (James Chiang) 5155602 Move argparse() to init() (JamesC) 2ab0146 Move assert num_nodes is set into main() (JamesC) 614c645 Clear TestNode objects after shutdown (JamesC) 6f40820 Add closing and flushing of logging handlers (JamesC) 6b71241 Refactor TestFramework main() into setup/shutdown (JamesC) ede8b76 Remove network_event_loop instance in close() (JamesC) Pull request description: This PR refactors BitcoinTestFramework to encapsulate setup and shutdown logic into dedicated methods, and adds a ~~TestWrapper~~ TestShell child class. This wrapper allows the underlying BitcoinTestFramework to run _between user inputs_ in a REPL environment, such as a Jupyter notebook or any interactive Python3 interpreter. The ~~TestWrapper~~ TestShell is motivated by the opportunity to expose the test-framework as a prototyping and educational toolkit. Examples of code prototypes enabled by ~~TestWrapper~~ TestShell can be found in the Optech [Taproot/Schnorr](https://github.com/bitcoinops/taproot-workshop) workshop repository. Usage example: ``` >>> import sys >>> sys.path.insert(0, "/path/to/bitcoin/test/functional") ``` ``` >>> from test_framework.test_wrapper import TestShell >>> test = TestShell() >>> test.setup(num_nodes=2) 20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Initializing test directory /path/to/bitcoin_func_test_XXXXXXX ``` ``` >>> test.nodes[0].generate(101) >>> test.nodes[0].getblockchaininfo()["blocks"] 101 ``` ``` >>> test.shutdown() 20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Stopping nodes 20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Cleaning up /path/to/bitcoin_func_test_XXXXXXX on exit 20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Tests successful ``` **Overview of changes to BitcoinTestFramework:** - Code moved to `setup()/shutdown()` methods. - Argument parsing logic encapsulated by `parse_args` method. - Success state moved to `BitcoinTestFramework.success`. _During Shutdown_ - `BitcoinTestFramework` logging handlers are flushed and removed. - `BitcoinTestFrameowork.nodes` list is cleared. - `NetworkThread.network_event_loop` is reset. (NetworkThread class). **Behavioural changes:** - Test parameters can now also be set when overriding BitcoinTestFramework.setup() in addition to overriding `set_test_params` method. - Potential exceptions raised in BitcoinTestFramework.setup() will be handled in main(). **Added files:** - ~~test_wrapper.py~~ `test_shell.py` - ~~test-wrapper.md~~ `test-shell.md` ACKs for top commit: jamesob: ACK bitcoin/bitcoin@19139ee jonatack: ACK 19139ee jnewbery: Rather than invalidate the three ACKs for a minor nit, can you force push back to 19139ee please? I think this PR was ready to merge before your last force push. jachiang: > Rather than invalidate the three ACKs for a minor nit, can you force push back to [19139ee](bitcoin/bitcoin@19139ee) please? I think this PR was ready to merge before your last force push. jnewbery: ACK 19139ee Tree-SHA512: 0c24f405f295a8580a9c8f1b9e0182b5d753eb08cc331424616dd50a062fb773d3719db4d08943365b1f42ccb965cc363b4bcc5beae27ac90b3460b349ed46b2
2 parents 33b155f + 19139ee commit bc38bb9

File tree

5 files changed

+337
-33
lines changed

5 files changed

+337
-33
lines changed

‎test/functional/README.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,16 @@ P2PInterface object and override the callback methods.
9999
Examples tests are [p2p_unrequested_blocks.py](p2p_unrequested_blocks.py),
100100
[p2p_compactblocks.py](p2p_compactblocks.py).
101101

102+
#### Prototyping tests
103+
104+
The [`TestShell`](test-shell.md) class exposes the BitcoinTestFramework
105+
functionality to interactive Python3 environments and can be used to prototype
106+
tests. This may be especially useful in a REPL environment with session logging
107+
utilities, such as
108+
[IPython](https://ipython.readthedocs.io/en/stable/interactive/reference.html#session-logging-and-restoring).
109+
The logs of such interactive sessions can later be adapted into permanent test
110+
cases.
111+
102112
### Test framework modules
103113
The following are useful modules for test developers. They are located in
104114
[test/functional/test_framework/](test_framework).

‎test/functional/test-shell.md

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
Test Shell for Interactive Environments
2+
=========================================
3+
4+
This document describes how to use the `TestShell` submodule in the functional
5+
test suite.
6+
7+
The `TestShell` submodule extends the `BitcoinTestFramework` functionality to
8+
external interactive environments for prototyping and educational purposes. Just
9+
like `BitcoinTestFramework`, the `TestShell` allows the user to:
10+
11+
* Manage regtest bitcoind subprocesses.
12+
* Access RPC interfaces of the underlying bitcoind instances.
13+
* Log events to the functional test logging utility.
14+
15+
The `TestShell` can be useful in interactive environments where it is necessary
16+
to extend the object lifetime of the underlying `BitcoinTestFramework` between
17+
user inputs. Such environments include the Python3 command line interpreter or
18+
[Jupyter](https://jupyter.org/) notebooks running a Python3 kernel.
19+
20+
## 1. Requirements
21+
22+
* Python3
23+
* `bitcoind` built in the same repository as the `TestShell`.
24+
25+
## 2. Importing `TestShell` from the Bitcoin Core repository
26+
27+
We can import the `TestShell` by adding the path of the Bitcoin Core
28+
`test_framework` module to the beginning of the PATH variable, and then
29+
importing the `TestShell` class from the `test_shell` sub-package.
30+
31+
```
32+
>>> import sys
33+
>>> sys.path.insert(0, "/path/to/bitcoin/test/functional")
34+
>>> from test_framework.test_shell import `TestShell`
35+
```
36+
37+
The following `TestShell` methods manage the lifetime of the underlying bitcoind
38+
processes and logging utilities.
39+
40+
* `TestShell.setup()`
41+
* `TestShell.shutdown()`
42+
43+
The `TestShell` inherits all `BitcoinTestFramework` members and methods, such
44+
as:
45+
* `TestShell.nodes[index].rpc_method()`
46+
* `TestShell.log.info("Custom log message")`
47+
48+
The following sections demonstrate how to initialize, run, and shut down a
49+
`TestShell` object.
50+
51+
## 3. Initializing a `TestShell` object
52+
53+
```
54+
>>> test = TestShell()
55+
>>> test.setup(num_nodes=2, setup_clean_chain=True)
56+
20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Initializing test directory /path/to/bitcoin_func_test_XXXXXXX
57+
```
58+
The `TestShell` forwards all functional test parameters of the parent
59+
`BitcoinTestFramework` object. The full set of argument keywords which can be
60+
used to initialize the `TestShell` can be found in [section
61+
#6](#custom-testshell-parameters) of this document.
62+
63+
**Note: Running multiple instances of `TestShell` is not allowed.** Running a
64+
single process also ensures that logging remains consolidated in the same
65+
temporary folder. If you need more bitcoind nodes than set by default (1),
66+
simply increase the `num_nodes` parameter during setup.
67+
68+
```
69+
>>> test2 = TestShell()
70+
>>> test2.setup()
71+
TestShell is already running!
72+
```
73+
74+
## 4. Interacting with the `TestShell`
75+
76+
Unlike the `BitcoinTestFramework` class, the `TestShell` keeps the underlying
77+
Bitcoind subprocesses (nodes) and logging utilities running until the user
78+
explicitly shuts down the `TestShell` object.
79+
80+
During the time between the `setup` and `shutdown` calls, all `bitcoind` node
81+
processes and `BitcoinTestFramework` convenience methods can be accessed
82+
interactively.
83+
84+
**Example: Mining a regtest chain**
85+
86+
By default, the `TestShell` nodes are initialized with a clean chain. This means
87+
that each node of the `TestShell` is initialized with a block height of 0.
88+
89+
```
90+
>>> test.nodes[0].getblockchaininfo()["blocks"]
91+
0
92+
```
93+
94+
We now let the first node generate 101 regtest blocks, and direct the coinbase
95+
rewards to a wallet address owned by the mining node.
96+
97+
```
98+
>>> address = test.nodes[0].getnewaddress()
99+
>>> test.nodes[0].generatetoaddress(101, address)
100+
['2b98dd0044aae6f1cca7f88a0acf366a4bfe053c7f7b00da3c0d115f03d67efb', ...
101+
```
102+
Since the two nodes are both initialized by default to establish an outbound
103+
connection to each other during `setup`, the second node's chain will include
104+
the mined blocks as soon as they propagate.
105+
106+
```
107+
>>> test.nodes[1].getblockchaininfo()["blocks"]
108+
101
109+
```
110+
The block rewards from the first block are now spendable by the wallet of the
111+
first node.
112+
113+
```
114+
>>> test.nodes[0].getbalance()
115+
Decimal('50.00000000')
116+
```
117+
118+
We can also log custom events to the logger.
119+
120+
```
121+
>>> test.nodes[0].log.info("Successfully mined regtest chain!")
122+
20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework.node0 (INFO): Successfully mined regtest chain!
123+
```
124+
125+
**Note: Please also consider the functional test
126+
[readme](../test/functional/README.md), which provides an overview of the
127+
test-framework**. Modules such as
128+
[key.py](../test/functional/test_framework/key.py),
129+
[script.py](../test/functional/test_framework/script.py) and
130+
[messages.py](../test/functional/test_framework/messages.py) are particularly
131+
useful in constructing objects which can be passed to the bitcoind nodes managed
132+
by a running `TestShell` object.
133+
134+
## 5. Shutting the `TestShell` down
135+
136+
Shutting down the `TestShell` will safely tear down all running bitcoind
137+
instances and remove all temporary data and logging directories.
138+
139+
```
140+
>>> test.shutdown()
141+
20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Stopping nodes
142+
20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Cleaning up /path/to/bitcoin_func_test_XXXXXXX on exit
143+
20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Tests successful
144+
```
145+
To prevent the logs from being removed after a shutdown, simply set the
146+
`TestShell.options.nocleanup` member to `True`.
147+
```
148+
>>> test.options.nocleanup = True
149+
>>> test.shutdown()
150+
20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Stopping nodes
151+
20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Not cleaning up dir /path/to/bitcoin_func_test_XXXXXXX on exit
152+
20XX-XX-XXTXX:XX:XX.XXXXXXX TestFramework (INFO): Tests successful
153+
```
154+
155+
The following utility consolidates logs from the bitcoind nodes and the
156+
underlying `BitcoinTestFramework`:
157+
158+
* `/path/to/bitcoin/test/functional/combine_logs.py
159+
'/path/to/bitcoin_func_test_XXXXXXX'`
160+
161+
## 6. Custom `TestShell` parameters
162+
163+
The `TestShell` object initializes with the default settings inherited from the
164+
`BitcoinTestFramework` class. The user can override these in
165+
`TestShell.setup(key=value)`.
166+
167+
**Note:** `TestShell.reset()` will reset test parameters to default values and
168+
can be called after the TestShell is shut down.
169+
170+
| Test parameter key | Default Value | Description |
171+
|---|---|---|
172+
| `bind_to_localhost_only` | `True` | Binds bitcoind RPC services to `127.0.0.1` if set to `True`.|
173+
| `cachedir` | `"/path/to/bitcoin/test/cache"` | Sets the bitcoind datadir directory. |
174+
| `chain` | `"regtest"` | Sets the chain-type for the underlying test bitcoind processes. |
175+
| `configfile` | `"/path/to/bitcoin/test/config.ini"` | Sets the location of the test framework config file. |
176+
| `coveragedir` | `None` | Records bitcoind RPC test coverage into this directory if set. |
177+
| `loglevel` | `INFO` | Logs events at this level and higher. Can be set to `DEBUG`, `INFO`, `WARNING`, `ERROR` or `CRITICAL`. |
178+
| `nocleanup` | `False` | Cleans up temporary test directory if set to `True` during `shutdown`. |
179+
| `noshutdown` | `False` | Does not stop bitcoind instances after `shutdown` if set to `True`. |
180+
| `num_nodes` | `1` | Sets the number of initialized bitcoind processes. |
181+
| `perf` | False | Profiles running nodes with `perf` for the duration of the test if set to `True`. |
182+
| `rpc_timeout` | `60` | Sets the RPC server timeout for the underlying bitcoind processes. |
183+
| `setup_clean_chain` | `False` | Initializes an empty blockchain by default. A 199-block-long chain is initialized if set to `True`. |
184+
| `randomseed` | Random Integer | `TestShell.options.randomseed` is a member of `TestShell` which can be accessed during a test to seed a random generator. User can override default with a constant value for reproducible test runs. |
185+
| `supports_cli` | `False` | Whether the bitcoin-cli utility is compiled and available for the test. |
186+
| `tmpdir` | `"/var/folders/.../"` | Sets directory for test logs. Will be deleted upon a successful test run unless `nocleanup` is set to `True` |
187+
| `trace_rpc` | `False` | Logs all RPC calls if set to `True`. |
188+
| `usecli` | `False` | Uses the bitcoin-cli interface for all bitcoind commands instead of directly calling the RPC server. Requires `supports_cli`. |

‎test/functional/test_framework/mininode.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -478,7 +478,8 @@ def close(self, timeout=10):
478478
wait_until(lambda: not self.network_event_loop.is_running(), timeout=timeout)
479479
self.network_event_loop.close()
480480
self.join(timeout)
481-
481+
# Safe to remove event loop.
482+
NetworkThread.network_event_loop = None
482483

483484
class P2PDataStore(P2PInterface):
484485
"""A P2P data store class.

‎test/functional/test_framework/test_framework.py

Lines changed: 62 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -99,12 +99,39 @@ def __init__(self):
9999
self.supports_cli = False
100100
self.bind_to_localhost_only = True
101101
self.set_test_params()
102-
103-
assert hasattr(self, "num_nodes"), "Test must set self.num_nodes in set_test_params()"
102+
self.parse_args()
104103

105104
def main(self):
106105
"""Main function. This should not be overridden by the subclass test scripts."""
107106

107+
assert hasattr(self, "num_nodes"), "Test must set self.num_nodes in set_test_params()"
108+
109+
try:
110+
self.setup()
111+
self.run_test()
112+
except JSONRPCException:
113+
self.log.exception("JSONRPC error")
114+
self.success = TestStatus.FAILED
115+
except SkipTest as e:
116+
self.log.warning("Test Skipped: %s" % e.message)
117+
self.success = TestStatus.SKIPPED
118+
except AssertionError:
119+
self.log.exception("Assertion failed")
120+
self.success = TestStatus.FAILED
121+
except KeyError:
122+
self.log.exception("Key error")
123+
self.success = TestStatus.FAILED
124+
except Exception:
125+
self.log.exception("Unexpected exception caught during testing")
126+
self.success = TestStatus.FAILED
127+
except KeyboardInterrupt:
128+
self.log.warning("Exiting after keyboard interrupt")
129+
self.success = TestStatus.FAILED
130+
finally:
131+
exit_code = self.shutdown()
132+
sys.exit(exit_code)
133+
134+
def parse_args(self):
108135
parser = argparse.ArgumentParser(usage="%(prog)s [options]")
109136
parser.add_argument("--nocleanup", dest="nocleanup", default=False, action="store_true",
110137
help="Leave bitcoinds and test.* datadir on exit or error")
@@ -135,6 +162,9 @@ def main(self):
135162
self.add_options(parser)
136163
self.options = parser.parse_args()
137164

165+
def setup(self):
166+
"""Call this method to start up the test framework object with options set."""
167+
138168
PortSeed.n = self.options.port_seed
139169

140170
check_json_precision()
@@ -181,33 +211,20 @@ def main(self):
181211
self.network_thread = NetworkThread()
182212
self.network_thread.start()
183213

184-
success = TestStatus.FAILED
214+
if self.options.usecli:
215+
if not self.supports_cli:
216+
raise SkipTest("--usecli specified but test does not support using CLI")
217+
self.skip_if_no_cli()
218+
self.skip_test_if_missing_module()
219+
self.setup_chain()
220+
self.setup_network()
185221

186-
try:
187-
if self.options.usecli:
188-
if not self.supports_cli:
189-
raise SkipTest("--usecli specified but test does not support using CLI")
190-
self.skip_if_no_cli()
191-
self.skip_test_if_missing_module()
192-
self.setup_chain()
193-
self.setup_network()
194-
self.run_test()
195-
success = TestStatus.PASSED
196-
except JSONRPCException:
197-
self.log.exception("JSONRPC error")
198-
except SkipTest as e:
199-
self.log.warning("Test Skipped: %s" % e.message)
200-
success = TestStatus.SKIPPED
201-
except AssertionError:
202-
self.log.exception("Assertion failed")
203-
except KeyError:
204-
self.log.exception("Key error")
205-
except Exception:
206-
self.log.exception("Unexpected exception caught during testing")
207-
except KeyboardInterrupt:
208-
self.log.warning("Exiting after keyboard interrupt")
222+
self.success = TestStatus.PASSED
209223

210-
if success == TestStatus.FAILED and self.options.pdbonfailure:
224+
def shutdown(self):
225+
"""Call this method to shut down the test framework object."""
226+
227+
if self.success == TestStatus.FAILED and self.options.pdbonfailure:
211228
print("Testcase failed. Attaching python debugger. Enter ? for help")
212229
pdb.set_trace()
213230

@@ -225,7 +242,7 @@ def main(self):
225242
should_clean_up = (
226243
not self.options.nocleanup and
227244
not self.options.noshutdown and
228-
success != TestStatus.FAILED and
245+
self.success != TestStatus.FAILED and
229246
not self.options.perf
230247
)
231248
if should_clean_up:
@@ -238,20 +255,33 @@ def main(self):
238255
self.log.warning("Not cleaning up dir {}".format(self.options.tmpdir))
239256
cleanup_tree_on_exit = False
240257

241-
if success == TestStatus.PASSED:
258+
if self.success == TestStatus.PASSED:
242259
self.log.info("Tests successful")
243260
exit_code = TEST_EXIT_PASSED
244-
elif success == TestStatus.SKIPPED:
261+
elif self.success == TestStatus.SKIPPED:
245262
self.log.info("Test skipped")
246263
exit_code = TEST_EXIT_SKIPPED
247264
else:
248265
self.log.error("Test failed. Test logging available at %s/test_framework.log", self.options.tmpdir)
249266
self.log.error("Hint: Call {} '{}' to consolidate all logs".format(os.path.normpath(os.path.dirname(os.path.realpath(__file__)) + "/../combine_logs.py"), self.options.tmpdir))
250267
exit_code = TEST_EXIT_FAILED
251-
logging.shutdown()
268+
# Logging.shutdown will not remove stream- and filehandlers, so we must
269+
# do it explicitly. Handlers are removed so the next test run can apply
270+
# different log handler settings.
271+
# See: https://docs.python.org/3/library/logging.html#logging.shutdown
272+
for h in list(self.log.handlers):
273+
h.flush()
274+
h.close()
275+
self.log.removeHandler(h)
276+
rpc_logger = logging.getLogger("BitcoinRPC")
277+
for h in list(rpc_logger.handlers):
278+
h.flush()
279+
rpc_logger.removeHandler(h)
252280
if cleanup_tree_on_exit:
253281
shutil.rmtree(self.options.tmpdir)
254-
sys.exit(exit_code)
282+
283+
self.nodes.clear()
284+
return exit_code
255285

256286
# Methods to override in subclass test scripts.
257287
def set_test_params(self):

0 commit comments

Comments
 (0)