Skip to content

SNOW-1939963: support query and dataframe options load#3097

Merged
sfc-gh-aling merged 33 commits intomainfrom
SNOW-1939963-feat-implement-query-and-prepared-query-option
Mar 3, 2025
Merged

SNOW-1939963: support query and dataframe options load#3097
sfc-gh-aling merged 33 commits intomainfrom
SNOW-1939963-feat-implement-query-and-prepared-query-option

Conversation

@sfc-gh-aling
Copy link
Contributor

@sfc-gh-aling sfc-gh-aling commented Feb 28, 2025 •
edited
Loading

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-NNNNNNN

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
    • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines

  3. Please describe how your code solves the related issue.

    Please write a short description of how your code change solves the related issue.

sfc-gh-yuwang and others added 27 commits February 25, 2025 15:53
@sfc-gh-yuwang @sfc-gh-aling
placeholder
1ea96c7
@sfc-gh-yuwang @sfc-gh-aling
SNOW-1863392:POC of python multi-process data ingestion (#2780)
5d8ce63
@sfc-gh-yuwang @sfc-gh-aling
SNOW-1887895: setup num-workers and allow creating temp table and add…
0674ba1 
… unit test (#2896)
@sfc-gh-yuwang @sfc-gh-aling
SNOW-1887892: improve process pool + thread pool logic, add retry log…
a8e51a9 
…ic, add timeout logic (#2924)
@sfc-gh-yuwang @sfc-gh-aling
SNOW-1887901: fix partitioning logic (#2944)
dab8d31
@sfc-gh-yuwang @sfc-gh-aling
SNOW-1848910:source data to parquet to snowflake data type mapping (#…
903d209 
…2949)
@sfc-gh-yuwang @sfc-gh-aling
SNOW-1887904:add batch fetching logic (#2970)
c7875bb
@sfc-gh-aling
SNOW-1904786: add telemetry to data source dbapi (#2979)
cd166f1
@sfc-gh-aling
SNOW-1916596: fix conn execute, make implmentation function private (#…
117a767 
…2991)
@sfc-gh-aling
SNOW-1887897: [DataSource] add custom schema support (#2993)
4af060f
@sfc-gh-yuwang @sfc-gh-aling
SNOW-1893781: support ingest oracle db (#2996)
620869b
@sfc-gh-yuwang @sfc-gh-jjiao @sfc-gh-aling
SNOW-1903096: add predicates (#3020)
8305124 
Co-authored-by: Jialin Jiao <jialin.jiao@snowflake.com>
@sfc-gh-aling
SNOW-1888002, SNOW-1923199: implement session_init_statement and impr…
99d9dd0 
…ove error experience (#3036)
@sfc-gh-aling
SNOW-1938435: implement fetch size based ingestion (#3050)
e9c2505
@sfc-gh-yuwang @sfc-gh-aling
SNOW-1933176: oracle perf improve (#3062)
3668f1a
@sfc-gh-aling @sfc-gh-yuwang
SNOW-1913915: add perf test scripts (#3064)
052d63a 
Co-authored-by: Yuyang Wang <yuyang.wang@snowflake.com>
@sfc-gh-aling
SNOW-1916054: release preparation (#3072)
0cdd6d5
@sfc-gh-aling
SNOW-1916054: release preparation update batch 2 (#3074)
cc9ea40
@sfc-gh-yuwang
NO SNOW: remove reduntant code (#3075)
b066860
@sfc-gh-yuwang
Merge branch 'main' into dev/data-source
ef4c007
@sfc-gh-yuwang
NO SNOW: merge into main preparation for dbapi (#3079)
df9719a
@sfc-gh-yuwang
Merge branch 'main' into dev/data-source
c5b77b0
@sfc-gh-aling
fix
74ddbee
@sfc-gh-aling
update schema inference
5fad43a
@sfc-gh-aling
update
c292891
@sfc-gh-aling
Merge branch 'dev/data-source' into NO-SNOW-fix-schema-inference
c994503
@sfc-gh-aling
fix test
5e9c139
@sfc-gh-aling sfc-gh-aling marked this pull request as ready for review February 28, 2025 17:36
@sfc-gh-aling sfc-gh-aling requested review from a team as code owners February 28, 2025 17:36
@sfc-gh-aling sfc-gh-aling requested review from sfc-gh-aalam, sfc-gh-jrose and sfc-gh-yuwang and removed request for a team February 28, 2025 17:36
@sfc-gh-aling sfc-gh-aling force-pushed the SNOW-1939963-feat-implement-query-and-prepared-query-option branch from 5e37b9f to 4909f5a Compare February 28, 2025 17:46
@sfc-gh-aling sfc-gh-aling changed the title SNOW-1939963: support query SNOW-1939963: support query and dataframe options load Feb 28, 2025
@sfc-gh-aling sfc-gh-aling mentioned this pull request Feb 28, 2025
Merged
8 tasks
sfc-gh-yixie
sfc-gh-yixie reviewed Mar 1, 2025
View reviewed changes
Base automatically changed from NO-SNOW-fix-schema-inference to main March 1, 2025 18:20
@sfc-gh-aling sfc-gh-aling requested a review from a team as a code owner March 1, 2025 18:21
@sfc-gh-aling sfc-gh-aling merged commit 32040bd into main Mar 3, 2025
42 of 137 checks passed
@sfc-gh-aling sfc-gh-aling deleted the SNOW-1939963-feat-implement-query-and-prepared-query-option branch March 3, 2025 17:42
@github-actions github-actions bot locked and limited conversation to collaborators Mar 3, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@sfc-gh-yixie sfc-gh-yixie sfc-gh-yixie approved these changes

@sfc-gh-yuwang sfc-gh-yuwang sfc-gh-yuwang approved these changes

@sfc-gh-aalam sfc-gh-aalam Awaiting requested review from sfc-gh-aalam sfc-gh-aalam is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-jrose sfc-gh-jrose Awaiting requested review from sfc-gh-jrose sfc-gh-jrose is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-nkumar sfc-gh-nkumar Awaiting requested review from sfc-gh-nkumar sfc-gh-nkumar was automatically assigned from snowflakedb/snowpark-pandas-api

@sfc-gh-joshi sfc-gh-joshi Awaiting requested review from sfc-gh-joshi sfc-gh-joshi was automatically assigned from snowflakedb/snowpark-pandas-api

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sfc-gh-aling @sfc-gh-yixie @sfc-gh-yuwang