Do not rewrite Spark views for Linkedin Spark 3.5#224
Merged
shanthoosh merged 2 commits intolinkedin:openhouse-1.5.2from Dec 17, 2025
Merged
Do not rewrite Spark views for Linkedin Spark 3.5#224shanthoosh merged 2 commits intolinkedin:openhouse-1.5.2from
shanthoosh merged 2 commits intolinkedin:openhouse-1.5.2from
Conversation
shanthoosh
approved these changes
Dec 16, 2025
shanthoosh
requested changes
Dec 17, 2025
.../scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala
Outdated
Show resolved
Hide resolved
shardulm94
approved these changes
Dec 17, 2025
shanthoosh
approved these changes
Dec 17, 2025
Collaborator
|
Thanks for fixing the failing tests in the build. Merging the PR. |
17 tasks
cbb330
pushed a commit
to linkedin/openhouse
that referenced
this pull request
Dec 19, 2025
## Summary This PR updates the iceberg version to pick up changes made in linkedin/iceberg#224 ## Changes - [x] Client-facing API Changes - [ ] Internal API Changes - [ ] Bug Fixes - [x] New Features - [ ] Performance Improvements - [ ] Code Style - [ ] Refactoring - [ ] Documentation - [ ] Tests For all the boxes checked, please include additional details of the changes made in this pull request. This introduces no breaking changes. By default, the newly added config `spark.sql.iceberg.rewriteViewCommands.enabled` is set to true so that the behavior matches exactly prior to this change too. ## Testing Done <!--- Check any relevant boxes with "x" --> - [ ] Manually Tested on local docker setup. Please include commands ran, and their output. - [ ] Added new tests for the changes made. - [ ] Updated existing tests to reflect the changes made. - [x] No tests added or updated. Please explain why. If unsure, please feel free to ask for help. - [ ] Some other form of testing like staging or soak time in production. Please explain. For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request. # Additional Information - [ ] Breaking Changes - [ ] Deprecations - [ ] Large PR broken into smaller PRs, and PR plan linked in the description. For all the boxes checked, include additional details of the changes made in this pull request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In Spark 3.5 along with
IcebergSparkSessionExtensions,CreateViewstatements in Spark lead to the creation ofCreateIcebergViewhere.This leads to the creation of
CreateV2ViewExechere.which in turn just calls
createViewon the catalog.createViewin Linkedin's internal session catalog is not implemented and hence the view creation does not happen.In Spark 3.1, we use HMS for handling views.
For Spark 3.5, we need a way to exclude this feature of rewriting views from taking place and let Spark handle it natively.
This PR, disables Iceberg from rewriting Spark view statements.