Skip to content

[ENH] t-SNE: Load openTSNE lazily #3883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
janezd merged 1 commit into from
Jun 14, 2019
Merged

[ENH] t-SNE: Load openTSNE lazily #3883

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 23 additions & 12 deletions Orange/projection/manifold.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,6 @@
from scipy.sparse.linalg import eigsh as arpack_eigh
import sklearn.manifold as skl_manifold

import openTSNE
import openTSNE.affinity
import openTSNE.initialization

import Orange
from Orange.data import Table, Domain, ContinuousVariable
from Orange.distance import Distance, DistanceModel, Euclidean
Expand All @@ -21,10 +17,6 @@
__all__ = ["MDS", "Isomap", "LocallyLinearEmbedding", "SpectralEmbedding",
"TSNE"]

# Disable t-SNE user warnings
openTSNE.tsne.log.setLevel(logging.ERROR)
openTSNE.affinity.log.setLevel(logging.ERROR)


def torgerson(distances, n_components=2, eigen_solver="auto"):
"""
Expand Down Expand Up @@ -190,6 +182,23 @@ def __init__(self, n_components=2, affinity='nearest_neighbors', gamma=None,
self.params = vars()


class lazy_openTSNE:
"""openTSNE uses numba, which is slow to load, so load it lazily."""
def __getattr__(self, item):
import sys
import openTSNE
import openTSNE.affinity
import openTSNE.initialization
if "openTSNE" in sys.modules:
Copy link
Contributor

@janezd janezd Jan 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pavlin-policar, I'm trying to revive this, but since the change caused a crash on travis in bioinformatics, it is difficult to debug.

Lookin for a cause, I found this condition very strange. When you import a module, it always appears in sys.modules, doesn't it? Why did you need this line, what did it try to prevent, what was the case when the condition was false?

Have you by any chance tried adding global openTSNE to the function?

Is it possible that bioinformatics tried to do something in parallel and this function was executed in parallel (though I don't see how this could have happened), and numba initialized twice? Qt, for instance, does (or used to) crash if you initialized QApplication twice.

# Disable t-SNE user warnings
openTSNE.tsne.log.setLevel(logging.ERROR)
openTSNE.affinity.log.setLevel(logging.ERROR)
return getattr(openTSNE, item)


openTSNE = lazy_openTSNE()


class TSNEModel(Projection):
"""A t-SNE embedding object. Supports further optimization as well as
adding new data into the existing embedding.
Expand All @@ -204,8 +213,8 @@ class TSNEModel(Projection):
pre_domain : Domain
Original data domain
"""
def __init__(self, embedding: openTSNE.TSNEEmbedding, table: Table,
pre_domain: Domain):
def __init__(self, embedding, table, pre_domain):
# type: (openTSNE.TSNEEmbedding, Table, Domain) -> None
transformer = TransformDomain(self)

def proj_variable(i):
Expand All @@ -221,7 +230,8 @@ def proj_variable(i):
class_vars=table.domain.class_vars,
metas=table.domain.metas)

def transform(self, X: np.ndarray, learning_rate=1, **kwargs) -> openTSNE.PartialTSNEEmbedding:
def transform(self, X, learning_rate=1, **kwargs):
# type: (np.ndarray, int, ...) -> openTSNE.PartialTSNEEmbedding
if sp.issparse(X):
raise TypeError(
"A sparse matrix was passed, but dense data is required. Use "
Expand Down Expand Up @@ -481,7 +491,8 @@ def prepare_embedding(self, affinities, initialization):
callbacks_every_iters=self.callbacks_every_iters,
)

def fit(self, X: np.ndarray, Y: np.ndarray = None) -> openTSNE.TSNEEmbedding:
def fit(self, X, Y=None):
# type: (np.ndarray, Optional[np.ndarray]) -> openTSNE.TSNEEmbedding
# Compute affinities and initial positions and prepare the embedding object
affinities = self.compute_affinities(X)
initialization = self.compute_initialization(X)
Expand Down