Skip to content

Commit 14cf702

Browse files
sfc-gh-azhansfc-gh-azhan
authored
SNOW-1491199 Generate Protobuf Python code for IR (#2518)
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1491199 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. - [x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: [Thread-safe Developer Guidelines](https://docs.google.com/document/d/162d_i4zZ2AfcGRXojj0jByt8EUq-DrSHPPnTa4QvwbA/edit#bookmark=id.e82u4nekq80k) 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. This PR removed Protobuf generated code from source and add logic to generate them when it is need. This pull request includes the addition of a script to install `protoc` and updates to multiple GitHub Actions workflows to use this script. Additionally, there is an update to the `CODEOWNERS` file. ### Installation and usage of `protoc`: * Added a new script `install_protoc.sh` to install `protoc` on GitHub Actions runners. This script handles downloading and installing the appropriate version of `protoc` based on the operating system and architecture. ### Code ownership: * Updated the `CODEOWNERS` file to add ownership for the `tests/unit/ast/` directory to the `@snowflakedb/snowpark-ir` team. ### Test passed Daily pandas precommit test: https://github.com/snowflakedb/snowpark-python/actions/runs/11645868643/job/32429396169 Daily notebook precommit test: https://github.com/snowflakedb/snowpark-python/actions/runs/11645871020 Daily precommit test: https://github.com/snowflakedb/snowpark-python/actions/runs/11645600647/job/32428853317 pandas Sproc precommit test: https://ci-dev-142.int.snowflakecomputing.com/job/SnowparkPandasStoredProcPrecommitTest/185/console ### Development Behavior Change After this PR, when developers try to pip install source code or use tox, they need to make sure `protoc` is installed. When pip install from wheel file, no `protoc` is required since the generated code should exist in the wheel file.
1 parent 6e95d34 commit 14cf702

File tree

18 files changed

+321
-744
lines changed

18 files changed

+321
-744
lines changed

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
/src/snowflake/snowpark/ @snowflakedb/snowpark-python-api-reviewers @snowflakedb/snowpark-ir
33
/src/snowflake/snowpark/modin/ @snowflakedb/snowpark-pandas-api
44
/tests/ast/ @snowflakedb/snowpark-ir
5+
/tests/unit/ast/ @snowflakedb/snowpark-ir
56
/tests/integ/modin/ @snowflakedb/snowpark-pandas-api
67
/tests/unit/modin/ @snowflakedb/snowpark-pandas-api
78
/docs/modin_api_coverage/ @snowflakedb/snowpark-pandas-api

‎.github/scripts/install_protoc.sh‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Copyright (c) 2024 Snowflake Computing Inc. All rights reserved.
4+
#
5+
6+
# This script is used to install protoc on Github actions.
7+
# Alternatively, we can use third party action like below but it requires a IT update
8+
# - name: Install Protoc
9+
# uses: arduino/setup-protoc@v3
10+
11+
set -eu
12+
13+
SCRIPT_NAME="$(basename "$0")"
14+
15+
echo "${SCRIPT_NAME} is running... "
16+
17+
PROTOC_VERSION=28.3
18+
PROTOC_OS_ARCH=""
19+
PROTOC_ZIP=""
20+
21+
buildProtocZIPName() {
22+
PROTOC_ZIP=protoc-${PROTOC_VERSION}-${PROTOC_OS_ARCH}.zip
23+
}
24+
25+
getOSNameAndArch(){
26+
KERNEL_TYPE=$(uname -s | tr '[:upper:]' '[:lower:]')
27+
ARCH=$(uname -m)
28+
echo "Your OS is ${KERNEL_TYPE} and arch is ${ARCH}"
29+
30+
case "${KERNEL_TYPE}" in
31+
linux)
32+
PROTOC_OS_ARCH="linux-x86_64"
33+
;;
34+
darwin)
35+
PROTOC_OS_ARCH="osx-universal_binary"
36+
;;
37+
mingw64* | msys* | cygwin*)
38+
PROTOC_OS_ARCH="win64"
39+
;;
40+
* )
41+
echo "Your Operating System ${KERNEL_TYPE} -> ITS NOT SUPPORTED"
42+
exit 1
43+
;;
44+
esac
45+
}
46+
47+
48+
downloadProtoc() {
49+
URL="https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/${PROTOC_ZIP}"
50+
51+
echo "Downloading ${PROTOC_ZIP} at ${URL}"
52+
53+
mkdir ${HOME}/local
54+
55+
curl -L -o "${PROTOC_ZIP}" "${URL}"
56+
unzip -o "${PROTOC_ZIP}" -d ${HOME}/local
57+
echo "$HOME/local/bin" >> $GITHUB_PATH
58+
}
59+
60+
61+
install() {
62+
getOSNameAndArch
63+
64+
buildProtocZIPName
65+
66+
downloadProtoc
67+
68+
export PATH="$HOME/local/bin:$PATH"
69+
echo "Protoc version: $(protoc --version)"
70+
}
71+
72+
install
73+
74+
echo "${SCRIPT_NAME} done."

‎.github/workflows/daily_jupyter_nb_test.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,9 @@ jobs:
2020
uses: actions/setup-python@v4
2121
with:
2222
python-version: '3.9'
23+
- name: Install protoc
24+
shell: bash
25+
run: .github/scripts/install_protoc.sh
2326
- name: Display Python version
2427
run: python -c "import sys; import os; print(\"\n\".join(os.environ[\"PATH\"].split(os.pathsep))); print(sys.version); print(sys.executable);"
2528
- name: Upgrade setuptools and pip
@@ -40,6 +43,9 @@ jobs:
4043
uses: actions/setup-python@v4
4144
with:
4245
python-version: 3.9
46+
- name: Install protoc
47+
shell: bash
48+
run: .github/scripts/install_protoc.sh
4349
- name: Upgrade setuptools and pip
4450
run: python -m pip install -U setuptools pip wheel
4551
- name: Generate wheel
@@ -82,6 +88,13 @@ jobs:
8288
env:
8389
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
8490
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
91+
- name: Install protoc
92+
shell: bash
93+
run: .github/scripts/install_protoc.sh
94+
- name: Add protoc to Windows path
95+
if: runner.os == 'Windows'
96+
run: |
97+
echo "$HOME/local/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
8598
- name: Download wheel(s)
8699
uses: actions/download-artifact@v4
87100
with:

‎.github/workflows/daily_modin_precommit.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@ jobs:
2828
run: python -c "import sys; import os; print(\"\n\".join(os.environ[\"PATH\"].split(os.pathsep))); print(sys.version); print(sys.executable);"
2929
- name: Upgrade setuptools and pip
3030
run: python -m pip install -U setuptools pip
31+
- name: Install protoc
32+
shell: bash
33+
run: .github/scripts/install_protoc.sh
3134
- name: Install tox
3235
run: python -m pip install tox
3336
- name: Run fix_lint
@@ -44,6 +47,9 @@ jobs:
4447
uses: actions/setup-python@v4
4548
with:
4649
python-version: 3.9
50+
- name: Install protoc
51+
shell: bash
52+
run: .github/scripts/install_protoc.sh
4753
- name: Upgrade setuptools and pip
4854
run: python -m pip install -U setuptools pip wheel
4955
- name: Generate wheel
@@ -67,6 +73,9 @@ jobs:
6773
uses: actions/setup-python@v4
6874
with:
6975
python-version: 3.8
76+
- name: Install protoc
77+
shell: bash
78+
run: .github/scripts/install_protoc.sh
7079
- name: Upgrade setuptools and pip
7180
run: python -m pip install -U setuptools pip
7281
- name: Install tox
@@ -91,6 +100,9 @@ jobs:
91100
env:
92101
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
93102
CLOUD_PROVIDER: "aws"
103+
- name: Install protoc
104+
shell: bash
105+
run: .github/scripts/install_protoc.sh
94106
- name: Upgrade setuptools and pip
95107
run: python -m pip install -U setuptools pip
96108
- name: Install tox
@@ -129,6 +141,13 @@ jobs:
129141
env:
130142
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
131143
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
144+
- name: Install protoc
145+
shell: bash
146+
run: .github/scripts/install_protoc.sh
147+
- name: Add protoc to Windows path
148+
if: runner.os == 'Windows'
149+
run: |
150+
echo "$HOME/local/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
132151
- name: Download wheel(s)
133152
uses: actions/download-artifact@v4
134153
with:
@@ -201,6 +220,9 @@ jobs:
201220
env:
202221
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
203222
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
223+
- name: Install protoc
224+
shell: bash
225+
run: .github/scripts/install_protoc.sh
204226
- name: Download wheel(s)
205227
uses: actions/download-artifact@v4
206228
with:
@@ -262,6 +284,9 @@ jobs:
262284
env:
263285
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
264286
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
287+
- name: Install protoc
288+
shell: bash
289+
run: .github/scripts/install_protoc.sh
265290
- name: Download wheel(s)
266291
uses: actions/download-artifact@v4
267292
with:
@@ -331,6 +356,9 @@ jobs:
331356
env:
332357
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
333358
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
359+
- name: Install protoc
360+
shell: bash
361+
run: .github/scripts/install_protoc.sh
334362
- name: Download wheel(s)
335363
uses: actions/download-artifact@v4
336364
with:
@@ -392,6 +420,9 @@ jobs:
392420
python-version: '3.9'
393421
- name: Display Python version
394422
run: python -c "import sys; print(sys.version)"
423+
- name: Install protoc
424+
shell: bash
425+
run: .github/scripts/install_protoc.sh
395426
- name: Upgrade setuptools and pip
396427
run: python -m pip install -U setuptools pip
397428
- name: Install tox
@@ -440,6 +471,9 @@ jobs:
440471
python-version: '3.9'
441472
- name: Upgrade setuptools and pip
442473
run: python -m pip install -U setuptools pip
474+
- name: Install protoc
475+
shell: bash
476+
run: .github/scripts/install_protoc.sh
443477
- name: Install Snowpark
444478
run: python -m pip install ".[modin-development]"
445479
- name: Install Sphinx

‎.github/workflows/daily_precommit.yml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ jobs:
3131
run: python -m pip install -U setuptools pip
3232
- name: Install tox
3333
run: python -m pip install tox
34+
- name: Install protoc
35+
shell: bash
36+
run: .github/scripts/install_protoc.sh
3437
- name: Run fix_lint
3538
run: python -m tox -e fix_lint
3639

@@ -63,6 +66,9 @@ jobs:
6366
uses: actions/setup-python@v4
6467
with:
6568
python-version: 3.9
69+
- name: Install protoc
70+
shell: bash
71+
run: .github/scripts/install_protoc.sh
6672
- name: Upgrade setuptools and pip
6773
run: python -m pip install -U setuptools pip wheel
6874
- name: Generate wheel
@@ -106,6 +112,13 @@ jobs:
106112
env:
107113
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
108114
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
115+
- name: Install protoc
116+
shell: bash
117+
run: .github/scripts/install_protoc.sh
118+
- name: Add protoc to Windows path
119+
if: runner.os == 'Windows'
120+
run: |
121+
echo "$HOME/local/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
109122
- name: Download wheel(s)
110123
uses: actions/download-artifact@v4
111124
with:
@@ -218,6 +231,9 @@ jobs:
218231
env:
219232
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
220233
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
234+
- name: Install protoc
235+
shell: bash
236+
run: .github/scripts/install_protoc.sh
221237
- name: Download wheel(s)
222238
uses: actions/download-artifact@v4
223239
with:
@@ -287,6 +303,9 @@ jobs:
287303
env:
288304
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
289305
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
306+
- name: Install protoc
307+
shell: bash
308+
run: .github/scripts/install_protoc.sh
290309
- name: Download wheel(s)
291310
uses: actions/download-artifact@v4
292311
with:
@@ -349,6 +368,13 @@ jobs:
349368
env:
350369
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
351370
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
371+
- name: Install protoc
372+
shell: bash
373+
run: .github/scripts/install_protoc.sh
374+
- name: Add protoc to Windows path
375+
if: runner.os == 'Windows'
376+
run: |
377+
echo "$HOME/local/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
352378
- name: Download wheel(s)
353379
uses: actions/download-artifact@v4
354380
with:
@@ -410,6 +436,9 @@ jobs:
410436
env:
411437
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
412438
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
439+
- name: Install protoc
440+
shell: bash
441+
run: .github/scripts/install_protoc.sh
413442
- name: Download wheel(s)
414443
uses: actions/download-artifact@v4
415444
with:
@@ -477,6 +506,9 @@ jobs:
477506
env:
478507
PARAMETER_PASSWORD: ${{ secrets.PARAMETER_PASSWORD }}
479508
CLOUD_PROVIDER: ${{ matrix.cloud-provider }}
509+
- name: Install protoc
510+
shell: bash
511+
run: .github/scripts/install_protoc.sh
480512
- name: Download wheel(s)
481513
uses: actions/download-artifact@v4
482514
with:
@@ -534,6 +566,9 @@ jobs:
534566
python-version: '3.9'
535567
- name: Display Python version
536568
run: python -c "import sys; print(sys.version)"
569+
- name: Install protoc
570+
shell: bash
571+
run: .github/scripts/install_protoc.sh
537572
- name: Upgrade setuptools and pip
538573
run: python -m pip install -U setuptools pip
539574
- name: Install tox
@@ -582,6 +617,9 @@ jobs:
582617
python-version: '3.9'
583618
- name: Upgrade setuptools and pip
584619
run: python -m pip install -U setuptools pip
620+
- name: Install protoc
621+
shell: bash
622+
run: .github/scripts/install_protoc.sh
585623
- name: Install Snowpark
586624
run: python -m pip install ".[modin-development]"
587625
- name: Install Sphinx

0 commit comments

Comments
 (0)