Commit ae1d7c7
feat: Add cancel and status queries to server-side async execution (#192)
* Added async_execution to async_db/cursor and db/cursor. Added an error, . Added a slot and getter for query_id to BaseCursor. (Temporarily?) edited setup.cfg to ignore flake8 C901: function is too complex.
* Removed ignore C901 from flake8 settings in setup.cfg.
* Fixed a couple of missing arguments in async_db/cursor.py on execute and execute_many calls.
* mypy and black cleanup.
* Removed set_parameters argument from all(?) functions. Started adding logic for SET async_execution.
* Added a bunch of callbacks to cursor tests.
* Pulled out a couple more set_parameters variables from function signatures. Edited callbacks and various tiny things in async test_cursor.py.
* Added, and commented out, server_side_async_url to unit/conftests.py.
* Removed test_set_parameters() from tests/async/cursor.py.
* Removed test_set_parameters() from tests/async/cursor.py.
* Added ability to see which exectute is failing (execute or executemany) on a unit test run.
* Added more explicit error messages to async and sync test_cursor.py modules.
* Added some periods.
* Replaced cursor reset call in async _do_execute().
* Updated query/message tuple decomposition in test_cursor to be more human-readable.
* Fixed a typo and function signature for db/test_cursor_server_side_async_execute() to remove async.
* Removed second hard-coding of query_id in server-side async id callback.
* Needed to add an await to an _api_request() call.
* Used InternalError to error out on no response to async server-side query. Changed AsyncExecutionUnavailableError to generically accept messages.
* Added additional checks on rowcount and description in test_cursor_server_side_async_execute().
* Added QueryResponse class.
* Minor changes requested on PR.
* Added an OperationalError is asynchronous query response is missing query_id.
* Had a typo.
* Added a warning if asyc_execution is set via a SET parameter rather than being sent in as an argument to execute(). Moved set parameter validation to its own function to deal with flake8 complaints re _do_exectute() being too complex.
* Started adding test_cursor_async_execute_error().
* Updated query_url argument in test_cursor_async_execute_error().
* Added AsyncExecutionUnavailableError on server-side async query execution for multi-statement queries.
* Seem to have dealt with auth issues in test_cursor_async_execute_error, but getting invalid set parameter on use_standard_sql. Also added # noqa: C901 to parse_type() definition in _types.py because flake8 was suddenly freaking out about it.
* Added all necessary set params to url string in test_cursor_async_execute_error().
* Cleaned up string input in test_cursor_async_execute_error().
* Now no token error.
* Multi-statement queries now error out correctly.
* Reworked a string to try to get commit/push to work.
* Had to add an extra auth callback to get all cursor.execute() calls to work. All error tests for async_execution should now be tested correctly.
* Removed some parameters from various fns in unit/async_db/test_cursor that I noticed were extraneous. Too bad mypy or flake8 isn't catching these :-/.
* Added error check for missing query_id on async_execution. A little bit of cleanup, also Black seems to have made a change or two.
* Fixed error for empty response.json on asynch execution. Also changed the use of SET async_execution to an error and included test for that error.
* Fixed error for empty response.json on asynch execution. Also changed the use of SET async_execution to an error and included test for that error.
* Fixed error for empty response.json on asynch execution. Also changed the use of SET async_execution to an error and included test for that error.
* Added a test to check that an server-side asynchronous execution returns a string, as a non-sync-execution query would return rowcount as an int.
* Added an integration test to check that an server-side asynchronous execution returns a string, as a non-sync-execution query would return rowcount as an int.~
* Added cancel() to async/cursor.py. Also fixed an error where I was getting empty query ids back from server-side async exectutions, and added an error for that case.
* Forgot that I'd commented out most of test_cursor.py.
* Trying to get rid of coroutine 'BaseCursor.execute' was never awaited warning. Not yet successful.
* Added unit tests for cancel and cancel errors.
* Fixed a mistake that would have failed the cancel() integration test.
* Fixed several imports that had disappeared (maybe during a merge?). Also fixed an error in test_ss_async_execution_cancel().
* get_status() and two unit tests are added. Integration test is failing with json that has correct field names but empty fields.
* Added a new QueryStatus, NOT_AVAILABLE, because checking status will return empty result the first few times. Fixed some issues with the unit tests and updated the integration tests for get_status().
* Added a comment.
* Updated a comment.
* Added stub fn for async execution fetch.
* Keep forgetting to uncomment test code and the pre-commit checks are removing imports. Left in some extra calls to time() and sleep() for now.
* Removed some extraneous testing code.
* Updated test_ss_async_execution_get_status() after Yoni pointed out that DDL operations will always return empty JSON. Now using an INSERT instead.
* Had to comment out test_ss_async_execution_get_status(), as it basically entered an infinite loop.
* Added ability to specify output_format in _api_request(), as status requests will fail if it is set.
* First set of requested changes on PR.
* Removed noqa on _do_execute().
* Renamed _find_async_problems() to _validate_ss_async_settings(). Removed test_cursor_server_side_async_cancel_error from integration tests.
* Moved call to _validate_ss_async_settings() into try.
* Added asyncio_mode=auto to pytest config in config.cfg, because I was tired of the continual warnings from pytest.
* Changed long query in test_queries_async integration tests. Paused execution of after cancel() to ensure I pick up the correct status message. Now sending output_format= on some calls to _api_request().
* Updated all unit tests that test SET parameters to not have output_format in the url.
* Changed query_loop() in integration tests/async/test_queries to check for more than one status before exiting the loop.
* Noticed that test_anyio_backend_import_issue() was commented out in sync/test_queries.py. That was done bc it won't run on my laptop, but it shouldn't have been committed that way.
* Added query tests to integration/dbapi/sync/test_queries.py. Changed async_exectution test to not count SET statements as queries when determining whether a query is multi-statment. Trying to get sleepEachRow() to work for long aync execution queries.
* Added query tests to integration/dbapi/sync/test_queries.py. Changed async_execution test to not count SET statements as queries when determining whether a query is multi-statment. Trying to get sleepEachRow() to work for long aync execution queries.
* Now errors out when use_standard_sql=0 rather than when it equals 1. This is because if it's off no log entries are written to query_history. Still using a long insert for integration tests on server-side async queries. Added and edited to unit tests for use_standard_sql correctness.
* Changed order of synchronous unit tests to move all server-side async tests to end.
* Changed order of asynchronous cursor unit tests to move all server-side async tests to end.
* Reordered integration and unit test modules to move all server-side async tests to end of modules to facilitate merging main.
* Moving JSON_OUTPUT_FORMAT outside of _api_request (#196)
* Updated docs to include information on server-side async query execution.
* Updated external table mention in comments and removed sentence in docs. Updated dictionary update in _api_request() to make mypy happy.
* Made a change to server-side execution explanation for clarity and to explain usefullness of that functionality.
* Renamed a function and moved table create and drop out of test_queries.py and into conftest.py. Currently name not defined error. Maybe it's in the wrong conftest file?
Co-authored-by: Petro Tiurin <[email protected]>1 parent 77471c2 commit ae1d7c7
File tree
10 files changed
+898
-237
lines changed- docsrc
- src/firebolt
- async_db
- db
- tests
- integration/dbapi
- async
- sync
- unit
- async_db
- db
10 files changed
+898
-237
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
| 18 | + | |
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
54 | 54 | | |
55 | | - | |
| 55 | + | |
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
64 | | - | |
| 64 | + | |
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
70 | | - | |
| 70 | + | |
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| |||
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
85 | | - | |
| 85 | + | |
86 | 86 | | |
87 | | - | |
| 87 | + | |
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
108 | | - | |
| 108 | + | |
109 | 109 | | |
110 | | - | |
| 110 | + | |
111 | 111 | | |
112 | 112 | | |
113 | | - | |
| 113 | + | |
114 | 114 | | |
115 | 115 | | |
116 | 116 | | |
117 | 117 | | |
118 | 118 | | |
119 | 119 | | |
120 | 120 | | |
121 | | - | |
| 121 | + | |
122 | 122 | | |
123 | | - | |
| 123 | + | |
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
129 | | - | |
130 | | - | |
131 | | - | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
132 | 132 | | |
133 | 133 | | |
134 | | - | |
| 134 | + | |
135 | 135 | | |
136 | | - | |
| 136 | + | |
137 | 137 | | |
138 | 138 | | |
139 | 139 | | |
| |||
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
148 | | - | |
| 148 | + | |
149 | 149 | | |
150 | | - | |
| 150 | + | |
151 | 151 | | |
152 | 152 | | |
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
156 | | - | |
| 156 | + | |
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
160 | 160 | | |
161 | 161 | | |
162 | 162 | | |
163 | 163 | | |
164 | | - | |
| 164 | + | |
165 | 165 | | |
166 | 166 | | |
167 | 167 | | |
| |||
181 | 181 | | |
182 | 182 | | |
183 | 183 | | |
184 | | - | |
| 184 | + | |
185 | 185 | | |
186 | 186 | | |
187 | 187 | | |
188 | 188 | | |
189 | | - | |
| 189 | + | |
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
193 | 193 | | |
194 | | - | |
| 194 | + | |
195 | 195 | | |
196 | 196 | | |
197 | 197 | | |
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
201 | 201 | | |
202 | | - | |
| 202 | + | |
203 | 203 | | |
204 | 204 | | |
205 | 205 | | |
| |||
216 | 216 | | |
217 | 217 | | |
218 | 218 | | |
219 | | - | |
220 | | - | |
| 219 | + | |
| 220 | + | |
221 | 221 | | |
222 | 222 | | |
223 | 223 | | |
| |||
231 | 231 | | |
232 | 232 | | |
233 | 233 | | |
234 | | - | |
| 234 | + | |
235 | 235 | | |
236 | 236 | | |
237 | 237 | | |
| |||
246 | 246 | | |
247 | 247 | | |
248 | 248 | | |
249 | | - | |
| 249 | + | |
250 | 250 | | |
251 | | - | |
| 251 | + | |
252 | 252 | | |
253 | 253 | | |
254 | 254 | | |
255 | 255 | | |
256 | | - | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
257 | 336 | | |
258 | | - | |
259 | 337 | | |
260 | 338 | | |
261 | 339 | | |
262 | | - | |
| 340 | + | |
263 | 341 | | |
264 | | - | |
| 342 | + | |
265 | 343 | | |
266 | | - | |
| 344 | + | |
267 | 345 | | |
268 | 346 | | |
269 | 347 | | |
270 | | - | |
| 348 | + | |
271 | 349 | | |
272 | | - | |
| 350 | + | |
273 | 351 | | |
274 | | - | |
| 352 | + | |
275 | 353 | | |
276 | 354 | | |
277 | 355 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| |||
0 commit comments