[BUG] `%%configure` Overwrites Entire session_configs Without Merge

[ChaonengQuan]

Describe the bug
The %%configure magic command in SparkMagic completely replaces the session_configs object with the JSON provided in the cell. It does not merge the new configuration with existing settings from ~/.sparkmagic/config.json, which can result in unintentional omission of important parameters like proxyUser or default Spark configs.

To Reproduce

1. Define a ~/.sparkmagic/config.json file with the following:

"session_configs": {
  "proxyUser": "jeffrey.zzz",
  "conf": {
    "spark.sql.catalog.spark_catalog.type": "hive"
  }
}

2. Run the following in a notebook:

%%configure -f
{
  "conf": {
    "spark.executor.memory": "8g"
  }
}

Expected behavior
Expected that the new settings provided via %%configure would be merged with the existing session_configs, or at least that the documentation would explicitly warn that a full replacement occurs.

Screenshots
N/A

Versions:

• SparkMagic: 0.21.0
• Livy: (if known)
• Spark: (if known)

Additional context
This behavior stems from the override() function in configuration.py#L34, where:

overrides[config] = value

This line replaces the full session_configs object rather than merging it with prior contents.

It would be helpful to clarify this behavior in the documentation or consider enhancing the configuration logic to support merging, or at least offer a warning when this overwrite occurs.

Thanks for maintaining SparkMagic!

[dosubot]

You are correct: the current behavior of the %%configure magic command in SparkMagic is to completely overwrite the session_configs object with the JSON provided in the cell, rather than merging it with any existing settings from ~/.sparkmagic/config.json. This means that any parameters not explicitly included in the new configuration (such as proxyUser or default Spark configs) will be omitted from the session.

This overwrite behavior is confirmed both in the documentation and in the code. The documentation for %%configure states that using the -f argument will delete the current session and recreate it with the applied configurations, which implies a full replacement rather than a merge (example notebook). In the code, the override function in configuration.py directly replaces the configuration object without merging (source).

There is a _recursive_merge function in the codebase that could theoretically be used to merge dictionaries, but it is not currently used by the override function or the %%configure logic (source).

At this time, there is no explicit warning in the documentation about this overwrite behavior, nor are there any recent issues, discussions, or commits addressing configuration merging or documentation warnings.

If merging behavior is important for your workflow, you may need to manually merge your desired settings before passing them to %%configure, or ensure that all required parameters are included each time you use the command.

To continue talking to Dosu, mention @dosu-bot.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}