Skip to content

Improve notebooks UX for dataframes with large number of columnsΒ #1579

@koperagen

Description

@koperagen

Example:

%use dataframe
#%%
val df = (0..100_000).toDataFrame {
  "i" from { it }
    "test" from { it }
}
df
#%%
val res = df.pivot { i }.with { test }

Few things are unclear
Image

  1. You actually can use res variable in the next cell, only without typesafe wrapper - message should be improved
  2. Need a way to suppress this error, like that codegen callback shouldn't run for this variable
  3. Actual error message looks fixable, although i doubt even with this fixed we won't get OOM or other performance problem
Caused by: org.jetbrains.kotlinx.jupyter.exceptions.ReplCompilerException: class org.jetbrains.kotlin.com.intellij.psi.impl.file.PsiLargeTextFileImpl cannot be cast to class org.jetbrains.kotlin.psi.KtFile (org.jetbrains.kotlin.com.intellij.psi.impl.file.PsiLargeTextFileImpl and org.jetbrains.kotlin.psi.KtFile are in unnamed module of loader 'app')
	at org.jetbrains.kotlinx.jupyter.repl.impl.JupyterCompilerImpl.compileSync(JupyterCompilerImpl.kt:151)
	at org.jetbrains.kotlinx.jupyter.repl.impl.InternalEvaluatorImpl.eval(InternalEvaluatorImpl.kt:126)
	at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl.execute_W38Nk0s$lambda$0$1(CellExecutorImpl.kt:95)
	at org.jetbrains.kotlinx.jupyter.repl.impl.ReplForJupyterImpl.withHost(ReplForJupyterImpl.kt:730)
	at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl.execute-W38Nk0s(CellExecutorImpl.kt:93)
	at org.jetbrains.kotlinx.jupyter.repl.execution.CellExecutor.execute-W38Nk0s$default(CellExecutor.kt:14)
	at org.jetbrains.kotlinx.jupyter.repl.impl.CellExecutorImpl$ExecutionContext.execute(CellExecutorImpl.kt:307)
	at org.jetbrains.kotlinx.dataframe.jupyter.ExecuteKt.execute(execute.kt:15)

Another problem is table rendering. Worth to check how other handle it. Some ideas: cut number of displayed columns, automatically switch to HTML/text/transposed view (at least makes sense for DataRow)

Video_2025-11-20_15-19-38.mp4

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions