Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Jun 13, 2025

What changes were proposed in this pull request?

This PR aims to add dfs-read-write and localstack examples.

Why are the changes needed?

To provide the following examples.

  1. How to add additional packages
spark.jars.packages: "org.apache.hadoop:hadoop-aws:3.4.1"
spark.jars.ivy: "/tmp/.ivy2.5.2"
  1. How to use S3
spark.hadoop.fs.defaultFS: "..."
spark.hadoop.fs.s3a.endpoint: "..."
spark.hadoop.fs.s3a.path.style.access: "..."
spark.hadoop.fs.s3a.access.key: "..."
spark.hadoop.fs.s3a.secret.key: "..."

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually run.

$ kubectl apply -f examples/localstack.yml
$ kubectl apply -f examples/dfs-read-write.yaml
$ kubectl logs -f dfs-read-write-0-driver
...
Success! Local Word Count 18 and DFS Word Count 18 agree.
...

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun
Copy link
Member Author

Thank you, @viirya . Merged to main.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-52467 branch June 13, 2025 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants