Add a User-Defined Table Function example #138
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lennartkats-db
force-pushed
the
add-udtf-example
branch
from
69e3edd to
12a7010
Compare
lennartkats-db
changed the title
- Add k-means clustering UDTF example for Unity Catalog - Focus documentation on UDTF pattern and SQL accessibility - Include Python implementation and SQL usage examples - Add CI/CD integration instructions
lennartkats-db
force-pushed
the
add-udtf-example
branch
from
ee6874e to
986dd79
Compare
denik
reviewed
Jan 12, 2026
| import csv | ||
| import os | ||
| except ImportError: | ||
| raise ImportError( |
There was a problem hiding this comment.
It's not great for debugging to lose original stack trace. Why not add the message but keep the stacktrace?
try:
...
except:
print(..., file.sys.stderr)
raise
| register_udtf_job: | ||
| name: register_udtf_job | ||
| schedule: | ||
| quartz_cron_expression: '0 0 8 * * ?' # daily at 8am |
There was a problem hiding this comment.
q - why do you need to register the same udtf daily?
There was a problem hiding this comment.
related q - how to unregister? is it just removing this job and there is some clean up process?
| if not self.rows: | ||
| return | ||
|
|
||
| import numpy as np |
There was a problem hiding this comment.
Could you add a comment why imports are here and not at the top level? Is it so that registration works without these libraries?
| def register(catalog: str, schema: str, name: str = "k_means"): | ||
| """Register k_means UDTF in Unity Catalog""" | ||
| spark = SparkSession.builder.getOrCreate() | ||
| source = inspect.getsource(SklearnKMeans) |
There was a problem hiding this comment.
How does this function gets access to dependencies? Does it rely on sklearn being pre-provisioned?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds an example for defining a User-Defined Table Function (UDTF) in Unity Catalog.
Highlighted files:
src/kmeans_udtf.py: registers the UDTFtests/test_kmeans_udtf.py: unit test for registration and usage (run withuv run pytest)resources/register_udtf_job.yml: job that registers the UDTF