Add torch array support for the variable explorer - #26001
Conversation
|
|
||
| if is_torch_tensor: | ||
| readonly = True | ||
| data = data.cpu().numpy() #data is converted to numpy array for display and edition, but the original tensor is not modified when accepting changes, so we can keep it read-only |
There was a problem hiding this comment.
Data is converted from torch.tensor to np.array. This is the main strategy to add support and reuse the already implemented features.
| if is_torch_tensor: | ||
| readonly = True | ||
| data = data.cpu().numpy() #data is converted to numpy array for display and edition, but the original tensor is not modified when accepting changes, so we can keep it read-only | ||
| data = data.detach().cpu().numpy() #data is converted to numpy array for display, but the original tensor is not modified when accepting changes, so we can keep it read-only |
There was a problem hiding this comment.
Added detach() to make sure that the torch tensor is not linked to the gradient calculation graph.
|
Thanks for your contribution @dfardila! You said:
Unfortunately, this is not a good strategy because it'll freeze Spyder when trying to explore large torch arrays. In other words, Spyder will stop working completely until the conversion is fully done, which wouldn't be a nice experience for our users. Instead, the conversion needs to be done on the kernel side, i.e. before sending a serialized copy of the object from spyder-kernels to Spyder. Since a kernel is a Python interpreter started in a separate process than Spyder, any computation taken place on it doesn't freeze Spyder itself. To understand how to implement that, please take a look at the way support for Polars dataframes was added in pull request #24558. And since you'll need to make a joint pull request in spyder-kernels to support torch arrays, please follow the instructions in our Contributing guide on how to do include your pull request there in a Spyder one. Let us know if you additional questions about that process or the implementation itself. |
2 participants
Introduction
Spyder is one of the favorite data science tools for academics and professionals around the world, thanks for its intuitive workflow, integrated debugging capabilities with IPython and its simple but yet powerful variable explorer.
In the recent years, the day-to-day libraries that a data scientist uses has expanded significantly, making the support of more data libraries an important feature for the spyder variable explorer.
With this in mind, I propose a simple solution to add torch arrays support for the variable explorer of spyder.
Description of Changes
The main strategy to add support for torch arrays consisted in converting the data to a numpy array early in the data verification process. This conversion is implemented in arrayeditor.py - line 941.
To verify if the array being explored is a torch array, the torch library needs to be imported in a safe way. This is done in arrayeditor.py - line 49 and collectionsdelegate.py - line 60. Not sure if the imports should go to the spyder kernels, if so please let me know.
The remaining changes manage the torch array before the conversion.
I changed the title of the variable explorer to from "Numpy Object Array" to "Object Array", so that it can be used for numpy or torch arrays.
Issue(s) Resolved
Fixes #7042
Affirmation
By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.
I certify the above statement is true and correct:
dfardila