Introduce SEA Backend Client with open and close Session functionality #576
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jacky Hu <[email protected]> Signed-off-by: Jesse Whitehouse <[email protected]> Co-authored-by: Jesse Whitehouse <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
…341) * Redact the URL query parameters from the urllib3.connectionpool logs Signed-off-by: Mubashir Kazia <[email protected]> * Fix code formatting Signed-off-by: Mubashir Kazia <[email protected]> * Add str check for the log record message arg dict values Signed-off-by: Mubashir Kazia <[email protected]> --------- Signed-off-by: Mubashir Kazia <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Jacky Hu <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* [PECO-1411] Support OAuth InHouse on GCP Signed-off-by: Jacky Hu <[email protected]> * Update changelog Signed-off-by: Jesse Whitehouse <[email protected]> --------- Signed-off-by: Jacky Hu <[email protected]> Signed-off-by: Jesse Whitehouse <[email protected]> Co-authored-by: Jesse Whitehouse <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* [PECO-1414] Support Databricks InHouse OAuth in Azure Signed-off-by: Jacky Hu <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Getting ready for test automation Signed-off-by: Ben Cassell <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* update owners Signed-off-by: yunbodeng-db <[email protected]> * update owners Signed-off-by: yunbodeng-db <[email protected]> * update owners Signed-off-by: yunbodeng-db <[email protected]> --------- Signed-off-by: yunbodeng-db <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Ben Cassell <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Jacky Hu <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* [PECO-1440] Expose current query id on cursor object Signed-off-by: Levko Kravets <[email protected]> * Clear `active_op_handle` when closing the cursor Signed-off-by: Levko Kravets <[email protected]> --------- Signed-off-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* Add a default for retry after Signed-off-by: Ben Cassell <[email protected]> * Applied black formatter Signed-off-by: Ben Cassell <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Set supports_native_boolean to True Signed-off-by: Alex Holyoke <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* Don't retry requests that fail with 404 Signed-off-by: Jesse Whitehouse <[email protected]> * Fix lint error Signed-off-by: Jesse Whitehouse <[email protected]> --------- Signed-off-by: Jesse Whitehouse <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* bump to 3.1.1 Signed-off-by: Ben Cassell <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* fix cookie setting Signed-off-by: Ben Cassell <[email protected]> * Removing cookie code Signed-off-by: Ben Cassell <[email protected]> --------- Signed-off-by: Ben Cassell <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* Create py.typed Signed-off-by: wyattscarpenter <[email protected]> * add -> Connection annotation Signed-off-by: wyattscarpenter <[email protected]> * massage the code to appease the particular version of the project's mypy deps Signed-off-by: wyattscarpenter <[email protected]> * fix circular import problem Signed-off-by: wyattscarpenter <[email protected]> --------- Signed-off-by: wyattscarpenter <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
fix the return types of the classes' __enter__ functions so that the type information is preserved in context managers eg with-as blocks Signed-off-by: wyattscarpenter <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Ben Cassell <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
changed authentication for proxy Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* Relax `pyarrow` pin Signed-off-by: Dave Hirschfeld <[email protected]> * Allow `pyarrow` 16 Signed-off-by: Dave Hirschfeld <[email protected]> * Update `poetry.lock` Signed-off-by: Dave Hirschfeld <[email protected]> --------- Signed-off-by: Dave Hirschfeld <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* Duplicate of applicable change from #93 Signed-off-by: Jesse Whitehouse <[email protected]> * Update changelog Signed-off-by: Jesse Whitehouse <[email protected]> * Fix after merge Signed-off-by: Levko Kravets <[email protected]> --------- Signed-off-by: Jesse Whitehouse <[email protected]> Signed-off-by: Levko Kravets <[email protected]> Co-authored-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* Enable `delta.feature.allowColumnDefaults` for all tables * Code style Signed-off-by: Levko Kravets <[email protected]> --------- Signed-off-by: Levko Kravets <[email protected]> Co-authored-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: Milan Lukac <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* Prepare release 3.2.0 Signed-off-by: Levko Kravets <[email protected]> * Update changelog Signed-off-by: Levko Kravets <[email protected]> --------- Signed-off-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* move py.typed to correct places https://peps.python.org/pep-0561/ says 'For namespace packages (see PEP 420), the py.typed file should be in the submodules of the namespace, to avoid conflicts and for clarity.'. Previously, when I added the py.typed file to this project, #382 , I was unaware this was a namespace package (although, curiously, it seems I had done it right initially and then changed to the wrong way). As PEP 561 warns us, this does create conflicts; other libraries in the databricks namespace package (such as, in my case, databricks-vectorsearch) are then treated as though they are typed, which they are not. This commit moves the py.typed file to the correct places, the submodule folders, fixing that problem. Signed-off-by: wyattscarpenter <[email protected]> * change target of mypy to src/databricks instead of src. I think this might fix the CI code-quality checks failure, but unfortunately I can't replicate that failure locally and the error message is unhelpful Signed-off-by: wyattscarpenter <[email protected]> * Possible workaround for bad error message 'error: --install-types failed (no mypy cache directory)'; see python/mypy#10768 (comment) Signed-off-by: wyattscarpenter <[email protected]> * fix invalid yaml syntax Signed-off-by: wyattscarpenter <[email protected]> * Best fix (#3) Fixes the problem by cding and supplying a flag to mypy (that mypy needs this flag is seemingly fixed/changed in later versions of mypy; but that's another pr altogether...). Also fixes a type error that was somehow in the arguments of the program (?!) (I guess this is because you guys are still using implicit optional) --------- Signed-off-by: wyattscarpenter <[email protected]> * return the old result_links default (#5) Return the old result_links default, make the type optional, & I'm pretty sure the original problem is that add_file_links can't take a None, so these statements should be in the body of the if-statement that ensures it is not None Signed-off-by: wyattscarpenter <[email protected]> * Update src/databricks/sql/utils.py "self.download_manager is unconditionally used later, so must be created. Looks this part of code is totally not covered with tests 🤔" Co-authored-by: Levko Kravets <[email protected]> Signed-off-by: wyattscarpenter <[email protected]> --------- Signed-off-by: wyattscarpenter <[email protected]> Co-authored-by: Levko Kravets <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
* Upgrade mypy This commit removes the flag (and cd step) from f53aa37 which we added to get mypy to treat namespaces correctly. This was apparently a bug in mypy, or behavior they decided to change. To get the new behavior, we must upgrade mypy. (This also allows us to remove a couple `# type: ignore` comment that are no longer needed.) This commit runs changes the version of mypy and runs `poetry lock`. It also conforms the whitespace of files in this project to the expectations of various tools and standard (namely: removing trailing whitespace as expected by git and enforcing the existence of one and only one newline at the end of a file as expected by unix and github.) It also uses https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade codebase due to a change in mypy behavior. For a similar reason, it also fixes a new type (or otherwise) errors: * "Return type 'Retry' of 'new' incompatible with return type 'DatabricksRetryPolicy' in supertype 'Retry'" * databricks/sql/auth/retry.py:225: error: object has no attribute update [attr-defined] * /test_param_escaper.py:31: DeprecationWarning: invalid escape sequence \) [as it happens, I think it was also wrong for the string not to be raw, because I'm pretty sure it wants all of its backslashed single-quotes to appear literally with the backslashes, which wasn't happening until now] * ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject [this is like a numpy version thing, which I fixed by being stricter about numpy version] --------- Signed-off-by: wyattscarpenter <[email protected]> * Incorporate suggestion. I decided the most expedient way of dealing with this type error was just adding the type ignore comment back in, but with a `[attr-defined]` specifier this time. I mean, otherwise I would have to restructure the code or figure out the proper types for a TypedDict for the dict and I don't think that's worth it at the moment. Signed-off-by: wyattscarpenter <[email protected]> --------- Signed-off-by: wyattscarpenter <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
ensure maintenance of current APIs of Connection while delegating responsibility Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
in case the openSession takes long, the initialisation of the session will not complete immediately. This could make the session attribute inaccessible. If the Connection is deleted in this time, the open() check will throw because the session attribute does not exist. Thus, we default to the Connection being closed in this case. This was not an issue before because open was a direct attribute of the Connection class. Caught in the integration tests. Signed-off-by: varun-edachali-dbx <[email protected]>
earlier, one of the integration tests was failing because 'session was not an attribute of Connection'. This is likely tied to a local configuration issue related to unittest that was causing an error in the test suite itself. The tests are now passing without checking for the session attribute. c676f9b Signed-off-by: varun-edachali-dbx <[email protected]>
This reverts commit d6b1b19. Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
* PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * PECOBLR-86 Improve logging for debug level Signed-off-by: Sai Shree Pradhan <[email protected]> * fixed format Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> * changed debug to error logs Signed-off-by: Sai Shree Pradhan <[email protected]> * used lazy logging Signed-off-by: Sai Shree Pradhan <[email protected]> --------- Signed-off-by: Sai Shree Pradhan <[email protected]> Signed-off-by: varun-edachali-dbx <[email protected]>
…couple-session" This reverts commit dbb2ec5, reversing changes made to 7192f11. Signed-off-by: varun-edachali-dbx <[email protected]>
…ecouple-session" This reverts commit bdb8381. Signed-off-by: varun-edachali-dbx <[email protected]>
ensures correctness of self.session.open call in Connection Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Signed-off-by: varun-edachali-dbx <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Description
Introduce the SEA Backend Client which re-uses the existing Thrift HTTP Client to make HTTP calls to the deployment. Add session opening and closure functionality and related tests.
How is this tested?
Related Tickets & Documents
https://docs.google.com/document/d/1Y-eXLhNqqhrMVGnOlG8sdFrCxBTN1GdQvuKG4IfHmo0/edit?tab=t.0#heading=h.4cz970y1mk93