Skip to content

[PRA-92] Add a context manager to create pre-configured Spark session#123

Merged
theoctober19th merged 15 commits intomainfrom
hackathon
Jan 26, 2026
Merged

[PRA-92] Add a context manager to create pre-configured Spark session#123
theoctober19th merged 15 commits intomainfrom
hackathon

Conversation

@theoctober19th
Copy link
Member

@theoctober19th theoctober19th commented Dec 13, 2024

This PR introduces a context manager SparkSession in spark8t.session which enables the users to create a pre-configured Spark session that comes with the following preconfigured:

  1. The K8s master
  2. The Spark Service account props
  3. The driver ip address
  4. Exemption of K8s master in the proxy settings (just in case it is being run on environments like PS7)

The objective here is to make it easy for users to use Spark inside the Spark pipeline. An example on how this can be used can be seen here.

Test mechanism:

  1. Unit tests are in place to ensure that the spark session calculates correct ip address for host, k8s master, does close spark session at the end of the session, and also adds k8s master to no_proxy environment variable.
  2. Integration tests are in place to ensure that this context manager can be used to run Spark jobs.
  3. Also tested in the Kubeflow <> Spark context here

@theoctober19th theoctober19th changed the title [Draft] Add a context manager for spark session [PRA-92] Add a context manager to create pre-configured Spark session Jan 21, 2026
@theoctober19th theoctober19th marked this pull request as ready for review January 21, 2026 14:29
Copy link
Contributor

@Batalex Batalex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm so happy that this feature is finally here!
For the longest time I would see this PR and think "oh this is neat, wonder when we will work on it again"
I left a few minor comments, but otherwise great job!

Copy link
Contributor

@Batalex Batalex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, we can leave the IPV6 question for later when we actually face the issue

Copy link
Collaborator

@welpaolo welpaolo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Great work Bikalpa

@theoctober19th theoctober19th merged commit c185da2 into main Jan 26, 2026
4 checks passed
@theoctober19th theoctober19th deleted the hackathon branch January 26, 2026 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants