Skip to content

Commit 7f2b10d

Browse files
sfc-gh-alingsfc-gh-aling
authored
SNOW-2336332: faster release automate fast path release package validation (#3788)
1 parent 3618118 commit 7f2b10d

File tree

4 files changed

+730
-0
lines changed

4 files changed

+730
-0
lines changed

‎scripts/conda_build_verification/README.md‎

Lines changed: 311 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,311 @@
1+
# Conda Build Verification
2+
3+
This directory contains scripts for Docker-based verification of Snowflake Snowpark Python conda packages across different architectures using the `continuumio/miniconda3` Docker image.
4+
5+
## Overview
6+
7+
The verification system automatically tests conda packages (.conda and .tar.bz2 formats) for **all Python versions 3.9-3.13** on both x86_64 and aarch64 architectures by:
8+
9+
1. Scanning the `package/` directory for conda packages in `linux-64`, `linux-aarch64`, and `noarch` subdirectories
10+
2. Launching Docker containers using `continuumio/miniconda3` (supports both architectures)
11+
3. Installing packages and their dependencies
12+
4. Running smoke tests with actual Snowflake connection to verify full functionality
13+
14+
**Note**: A valid `parameters.py` file with Snowflake connection parameters is required for all testing.
15+
16+
## Files
17+
18+
- `conda_build_verification.sh` - Main orchestration script
19+
- `docker_verify.sh` - Internal script that runs inside Docker containers
20+
- `smoke_test.py` - Smoke test script for package verification
21+
- `parameters.py` - **Required** Snowflake connection parameters - must be created manually
22+
- `package/` - Directory containing conda packages organized by architecture
23+
24+
## Package Structure
25+
26+
The verification expects packages to be organized as follows:
27+
```
28+
package/
29+
├── linux-64/ # x86_64 packages
30+
│ ├── snowflake-snowpark-python-1.38.0-py39_0.conda
31+
│ ├── snowflake-snowpark-python-1.38.0-py310_0.conda
32+
│ ├── snowflake-snowpark-python-1.38.0-py311_0.conda
33+
│ ├── snowflake-snowpark-python-1.38.0-py312_0.conda
34+
│ ├── snowflake-snowpark-python-1.38.0-py313_0.conda
35+
│ └── ... (corresponding .tar.bz2 files)
36+
├── linux-aarch64/ # aarch64 packages
37+
│ ├── snowflake-snowpark-python-1.38.0-py39_0.conda
38+
│ ├── snowflake-snowpark-python-1.38.0-py310_0.conda
39+
│ └── ... (all Python versions 3.9-3.13)
40+
└── noarch/ # Architecture-independent packages
41+
├── snowflake-snowpark-python-1.39.0-py39_0.conda
42+
├── snowflake-snowpark-python-1.39.0-py310_0.conda
43+
└── ... (all Python versions 3.9-3.13)
44+
```
45+
46+
## Usage
47+
48+
### Basic Usage
49+
50+
**Prerequisites**: Ensure you have created `parameters.py` with valid Snowflake connection parameters (see Required Setup section above).
51+
52+
```bash
53+
# Test all Python versions (3.9-3.13) on noarch only
54+
./conda_build_verification.sh
55+
56+
# Test all Python versions on specific architecture(s)
57+
./conda_build_verification.sh linux-64
58+
./conda_build_verification.sh linux-64 noarch
59+
./conda_build_verification.sh linux-64 linux-aarch64 noarch
60+
```
61+
62+
### Required Setup
63+
64+
**Before running any verification**, you need a `parameters.py` file with Snowflake connection parameters. You have two options:
65+
66+
#### Option 1: Automatic GPG Decryption (Recommended for CI/CD)
67+
If you have access to the encrypted parameters file and GPG key:
68+
```bash
69+
export GPG_KEY="your-gpg-passphrase"
70+
./conda_build_verification.sh # Automatically decrypts parameters.py.gpg
71+
```
72+
73+
#### Option 2: Manual Creation
74+
Manually create a `parameters.py` file in the `conda_build_verification/` directory:
75+
76+
```python
77+
# parameters.py
78+
CONNECTION_PARAMETERS = {
79+
"user": "your_username",
80+
"password": "your_password",
81+
"account": "your_account",
82+
"warehouse": "your_warehouse",
83+
"database": "your_database",
84+
"schema": "your_schema"
85+
}
86+
```
87+
88+
Then run verification:
89+
90+
```bash
91+
# Run verification on default architecture (noarch) for all Python versions
92+
./conda_build_verification.sh
93+
94+
# Run verification on specific architectures for all Python versions
95+
./conda_build_verification.sh linux-64 noarch
96+
```
97+
98+
### Command Line Syntax
99+
100+
```bash
101+
./conda_build_verification.sh [architecture...]
102+
```
103+
104+
**Parameters:**
105+
- `architecture...` (optional): One or more architecture directories to test
106+
- Available: `linux-64`, `linux-aarch64`, `noarch`
107+
- Default: `noarch` only if no architectures specified
108+
- **Python versions**: Automatically tests all Python versions 3.9-3.13
109+
110+
**Examples:**
111+
- `./conda_build_verification.sh` → Test all Python versions (3.9-3.13) on noarch
112+
- `./conda_build_verification.sh linux-64` → Test all Python versions on linux-64 only
113+
- `./conda_build_verification.sh linux-64 noarch` → Test all Python versions on both linux-64 and noarch
114+
- `./conda_build_verification.sh linux-64 linux-aarch64 noarch` → Test all Python versions on all architectures
115+
116+
### Environment Variables
117+
118+
- `GPG_KEY` - (Optional) Passphrase for automatically decrypting `scripts/parameters.py.gpg`
119+
120+
## Docker Requirements
121+
122+
- Docker must be installed and accessible
123+
- Docker daemon must be running
124+
- Internet access for pulling `continuumio/miniconda3` image
125+
126+
## How It Works
127+
128+
1. **Prerequisites Validation**:
129+
- Checks for Docker availability
130+
- Attempts GPG decryption of `parameters.py.gpg` if `GPG_KEY` environment variable is set
131+
- Validates `parameters.py` file existence and provides helpful error messages if missing
132+
- Fails fast before any Docker operations begin
133+
134+
2. **Package Discovery**: Scans `package/linux-64`, `package/linux-aarch64`, and `package/noarch` for packages across all Python versions (3.9-3.13)
135+
- **Strict Validation**: If packages are not found in any requested architecture directory, the script errors out immediately
136+
137+
3. **Docker Container Launch**: For each architecture with packages:
138+
- Launches `continuumio/miniconda3` container with appropriate `--platform` flag
139+
- **Cross-platform testing**: Can test aarch64 packages on x86_64 hosts and vice versa
140+
- Mounts verification scripts and package directory
141+
- Runs `docker_verify.sh` inside the container
142+
143+
4. **Package Testing**: Inside each container, for each Python version (3.9-3.13):
144+
- Creates isolated conda environment for the specific Python version
145+
- Installs package dependencies
146+
- Installs the conda package (.conda and .tar.bz2 formats) if available for that version
147+
- Runs smoke tests for that version
148+
- Reports results per Python version
149+
150+
5. **Smoke Testing**:
151+
- Connects to Snowflake using the `parameters.py` configuration
152+
- Runs a test query (`SELECT 1 as A`) to verify full functionality
153+
- Validates that the installed package can successfully interact with Snowflake
154+
155+
6. **Cleanup**: Removes containers and temporary files
156+
157+
## Architecture Support
158+
159+
The `continuumio/miniconda3` Docker image supports both:
160+
- **x86_64** (Intel/AMD 64-bit)
161+
- **aarch64** (ARM 64-bit, Apple Silicon, ARM servers)
162+
163+
Docker uses the `--platform` flag to specify the target architecture:
164+
- `linux-64` → `--platform linux/amd64` (x86_64)
165+
- `linux-aarch64` → `--platform linux/arm64` (aarch64)
166+
- `noarch` → `--platform linux/amd64` (x86_64 for compatibility)
167+
168+
## Example Output
169+
170+
```
171+
Testing Python versions: 3.9, 3.10, 3.11, 3.12, 3.13
172+
Using default architecture: noarch
173+
Testing architectures: noarch
174+
Script directory: /path/to/scripts/conda_build_verification
175+
Project root: /path/to/snowpark-python
176+
GPG_KEY found, attempting to decrypt parameters.py...
177+
✅ Successfully decrypted parameters.py
178+
Starting Docker-based conda package verification...
179+
Found packages in noarch
180+
=== Running verification for noarch ===
181+
Found packages in noarch, will test all Python versions (3.9-3.13)
182+
Running Docker command for noarch...
183+
184+
=== Testing Python 3.9 ===
185+
Testing conda package: snowflake-snowpark-python-1.39.0-py39_0.conda
186+
✅ Package test completed successfully: snowflake-snowpark-python-1.39.0-py39_0.conda
187+
Testing tar.bz2 package: snowflake-snowpark-python-1.39.0-py39_0.tar.bz2
188+
✅ Package test completed successfully: snowflake-snowpark-python-1.39.0-py39_0.tar.bz2
189+
✅ All tests passed for Python 3.9
190+
191+
=== Testing Python 3.10 ===
192+
Testing conda package: snowflake-snowpark-python-1.39.0-py310_0.conda
193+
✅ Package test completed successfully: snowflake-snowpark-python-1.39.0-py310_0.conda
194+
Testing tar.bz2 package: snowflake-snowpark-python-1.39.0-py310_0.tar.bz2
195+
✅ Package test completed successfully: snowflake-snowpark-python-1.39.0-py310_0.tar.bz2
196+
✅ All tests passed for Python 3.10
197+
198+
... (similar output for Python 3.11, 3.12, 3.13)
199+
200+
=== Final Results ===
201+
✅ All package tests completed successfully across all Python versions
202+
203+
✅ Verification successful for noarch
204+
Cleaning up...
205+
Removed decrypted parameters.py
206+
✅ All verifications completed successfully
207+
```
208+
209+
## Troubleshooting
210+
211+
### Common Issues
212+
213+
1. **Docker not found**
214+
```
215+
Error: Docker is not installed or not in PATH
216+
```
217+
Install Docker and ensure it's in your PATH.
218+
219+
2. **No packages found**
220+
```
221+
❌ Error: No packages found in requested architecture: linux-aarch64
222+
Available packages can be found in:
223+
- linux-64
224+
- noarch
225+
```
226+
This error occurs when the script cannot find packages for a requested architecture directory.
227+
228+
**To fix**: Ensure packages are in the correct `package/` subdirectories with proper naming patterns (e.g., `snowflake-snowpark-python-*-py39_*.conda`), or use one of the available architectures listed in the error message.
229+
230+
3. **GPG decryption fails**
231+
```
232+
❌ Error: Failed to decrypt parameters.py.gpg
233+
Please check your GPG_KEY environment variable or create parameters.py manually
234+
```
235+
This occurs when GPG decryption fails. Possible causes:
236+
- Incorrect `GPG_KEY` passphrase
237+
- GPG not installed or configured
238+
- Corrupted `.gpg` file
239+
240+
**To fix**: Verify the GPG_KEY value or create `parameters.py` manually.
241+
242+
4. **Missing parameters.py.gpg file**
243+
```
244+
❌ Error: parameters.py.gpg not found at /path/to/scripts/parameters.py.gpg
245+
```
246+
This occurs when `GPG_KEY` is set but the encrypted file doesn't exist.
247+
248+
**To fix**: Ensure the encrypted file exists at `scripts/parameters.py.gpg` or create `parameters.py` manually.
249+
250+
5. **Container startup fails**
251+
Check Docker daemon is running and you have internet access to pull the image.
252+
253+
6. **Missing parameters.py file**
254+
```
255+
❌ Error: Required file 'parameters.py' not found in /path/to/conda_build_verification/
256+
257+
The verification requires a parameters.py file with Snowflake connection parameters.
258+
Please create /path/to/conda_build_verification/parameters.py with the following format:
259+
260+
CONNECTION_PARAMETERS = {
261+
"user": "your_username",
262+
"password": "your_password",
263+
"account": "your_account",
264+
"warehouse": "your_warehouse",
265+
"database": "your_database",
266+
"schema": "your_schema"
267+
}
268+
```
269+
270+
This error occurs when `parameters.py` is missing from the verification directory. The script now checks for this file early and fails fast.
271+
272+
**To fix**: Create a `parameters.py` file in the `conda_build_verification/` directory with your connection parameters:
273+
```python
274+
CONNECTION_PARAMETERS = {
275+
"user": "your_username",
276+
"password": "your_password",
277+
"account": "your_account",
278+
"warehouse": "your_warehouse",
279+
"database": "your_database",
280+
"schema": "your_schema"
281+
}
282+
```
283+
284+
### Manual Testing
285+
286+
To manually test packages for all Python versions:
287+
288+
```bash
289+
# Run container interactively
290+
docker run -it --rm \
291+
-v $(pwd):/verification \
292+
-v $(pwd)/package/linux-64:/packages \
293+
-e ARCH_DIR=linux-64 \
294+
continuumio/miniconda3:latest \
295+
bash
296+
297+
# Inside container, run the verification script (tests all Python versions 3.9-3.13)
298+
bash /verification/docker_verify.sh
299+
```
300+
301+
## Dependencies
302+
303+
The verification installs these dependencies inside each test environment:
304+
- cloudpickle<=3.1.1
305+
- pyyaml
306+
- snowflake-connector-python
307+
- tzlocal
308+
- python-dateutil
309+
- protobuf
310+
311+
These match the dependencies expected by Snowflake Snowpark Python.

0 commit comments

Comments
 (0)