fix: Install kernel spec into venv instead of user directory (#29)

* fix: Install kernel spec into venv instead of user directory

Previously, ipykernel specs were installed with --user flag, placing them
in ~/Library/Jupyter/kernels/. This caused the Deepnote Jupyter server to
fall back to generic Python kernels because it couldn't discover the
venv-specific kernel spec.

This resulted in a mismatch where:
- Shell commands (!pip install) used the venv's Python (via PATH)
- But the kernel interpreter used system Python
- Leading to ModuleNotFoundError for packages installed in the venv

Now installing kernel specs with --prefix into the venv itself at
<venv>/share/jupyter/kernels/. Jupyter automatically discovers kernel
specs in this location when started with the venv's Python, ensuring
the correct interpreter is used.

Each .deepnote file maintains its own isolated venv with its own kernel
spec, preventing conflicts when multiple notebooks are open.

Author: Artmann

CLAUDE.md

@@ -1,6 +1,7 @@
 ## Code Style & Organization
 - Order method, fields and properties, first by accessibility and then by alphabetical order.
 - Don't add the Microsoft copyright header to new files.
+- Use `Uri.joinPath()` for constructing file paths to ensure platform-correct path separators (e.g., `Uri.joinPath(venvPath, 'share', 'jupyter', 'kernels')` instead of string concatenation with `/`)
 
 ## Testing
 - Unit tests use Mocha/Chai framework with `.unit.test.ts` extension

src/kernels/deepnote/deepnoteToolkitInstaller.node.ts

@@ -65,6 +65,8 @@ export class DeepnoteToolkitInstaller implements IDeepnoteToolkitInstaller {
         const venvPath = this.getVenvPath(deepnoteFileUri);
         const venvKey = venvPath.fsPath;
 
+        logger.info(`Ensuring virtual environment at ${venvKey}`);
+
         // Wait for any pending installation for this venv to complete
         const pendingInstall = this.pendingInstallations.get(venvKey);
         if (pendingInstall) {
@@ -210,6 +212,7 @@ export class DeepnoteToolkitInstaller implements IDeepnoteToolkitInstaller {
                 logger.info('deepnote-toolkit installed successfully in venv');
 
                 // Install kernel spec so the kernel uses this venv's Python
+                // Install into the venv itself (not --user) so the Deepnote server can discover it
                 logger.info('Installing kernel spec for venv...');
                 try {
                     // Reuse the process service with system environment
@@ -219,15 +222,23 @@ export class DeepnoteToolkitInstaller implements IDeepnoteToolkitInstaller {
                             '-m',
                             'ipykernel',
                             'install',
-                            '--user',
+                            '--prefix',
+                            venvPath.fsPath,
                             '--name',
                             `deepnote-venv-${this.getVenvHash(deepnoteFileUri)}`,
                             '--display-name',
                             `Deepnote (${this.getDisplayName(deepnoteFileUri)})`
                         ],
                         { throwOnStdErr: false }
                     );
-                    logger.info('Kernel spec installed successfully');
+                    const kernelSpecPath = Uri.joinPath(
+                        venvPath,
+                        'share',
+                        'jupyter',
+                        'kernels',
+                        `deepnote-venv-${this.getVenvHash(deepnoteFileUri)}`
+                    );
+                    logger.info(`Kernel spec installed successfully to ${kernelSpecPath.fsPath}`);
                 } catch (ex) {
                     logger.warn(`Failed to install kernel spec: ${ex}`);
                     // Don't fail the entire installation if kernel spec creation fails