Skip to content

Commit ddbcd96

Browse files
jmhsiehclaude
jmhsieh
and
claude
authored
Add dependency verification documentation (#110)
Add docs for diagnosing and resolving package version mismatches between local and Ray worker environments, including the compare_ray_environments tool and common issues/solutions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 563f084 commit ddbcd96

File tree

2 files changed

+341
-0
lines changed

2 files changed

+341
-0
lines changed

‎docs/docs.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,7 @@
168168
"geneva/jobs/startup",
169169
"geneva/jobs/materialized-views",
170170
"geneva/jobs/contexts",
171+
"geneva/deployment/dependency-verification",
171172
"geneva/jobs/console",
172173
"geneva/jobs/performance"
173174
]

‎docs/geneva/deployment/dependency-verification.mdx‎

Lines changed: 340 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,340 @@
1+
---
2+
title: Dependency Verification
3+
sidebarTitle: Dependency Verification
4+
description: Diagnose and resolve package version mismatches between local and Ray worker environments.
5+
icon: magnifying-glass-chart
6+
---
7+
8+
When running Geneva UDFs on Ray, your code is serialized locally and executed on remote workers. If the worker environment differs from your local environment, you may encounter subtle and difficult-to-debug errors.
9+
10+
## Example environment mismatch errors
11+
12+
| Symptom | Likely Cause |
13+
|---------|--------------|
14+
| `TypeError: Enum.__new__() missing 1 required positional argument` | `attrs` version mismatch |
15+
| `TypeError: Can't instantiate abstract class` | Package structure differences |
16+
| `ArrowInvalid: cannot cast` / serialization errors | NumPy 1.x vs 2.x mismatch |
17+
| `ModuleNotFoundError` on workers | Package only installed locally |
18+
| Model loading failures | PyTorch version mismatch |
19+
| Permission denied errors | Missing API keys in envrionment variables |
20+
21+
These issues are notoriously difficult to debug because the error messages often don't indicate the root cause.
22+
23+
24+
## The `compare_ray_environments` Tool
25+
26+
Geneva provides a diagnostic tool to compare your local environment against Ray workers.
27+
28+
If you are encountering a hang or exception you can use the following diagnosis worklflow to resolve the problem.
29+
30+
<Steps>
31+
<Step>
32+
**Run the diagnostic tool** programatically or via the CLI.
33+
</Step>
34+
<Step>
35+
**Check PACKAGES and ENV VARS output sections for mismatches**.
36+
</Step>
37+
<Step>
38+
**Identify critical packages**: numpy, torch, pyarrow, attrs, pydantic.
39+
</Step>
40+
<Step>
41+
**Identify inconsistent environment variables**: `AWS_*`, `GOOGLE_APPLICATION_CREDENTIALS`
42+
</Step>
43+
<Step>
44+
**Fix with manifest** for quick testing:
45+
```python
46+
from geneva.manifest.builder import GenevaManifestBuilder
47+
manifest = GenevaManifestBuilder.create("fix").pip(["numpy==1.26.4"]).build()
48+
```
49+
</Step>
50+
<Step>
51+
**OPTIONAL: Build custom image** for production (if using KubeRay).
52+
</Step>
53+
</Steps>
54+
55+
### Programmatic Usage
56+
57+
<CodeGroup>
58+
```python Python icon="python"
59+
from geneva.runners.ray.compare_env import compare_ray_environments
60+
61+
# Compare and print (requires Geneva context to be initialize via `with db.context(..)`)
62+
result = compare_ray_environments()
63+
64+
# Compare environments, filtering environment variables with specified prefix
65+
result = compare_ray_environments(env_prefix="PY")
66+
```
67+
</CodeGroup>
68+
69+
### CLI Usage
70+
71+
<CodeGroup>
72+
```bash CLI icon="terminal"
73+
# Connect to existing Ray cluster
74+
python -m geneva.runners.ray.compare_env
75+
76+
# Start new local Ray cluster
77+
python -m geneva.runners.ray.compare_env --address local
78+
79+
# Filter env vars by prefix
80+
python -m geneva.runners.ray.compare_env --env-prefix RAY
81+
82+
# Show full JSON snapshots
83+
python -m geneva.runners.ray.compare_env --show-all
84+
85+
# Skip sys.path comparison
86+
python -m geneva.runners.ray.compare_env --no-sys-path
87+
```
88+
</CodeGroup>
89+
90+
## Understanding the Output
91+
92+
The tool outputs several sections to help you identify mismatches.
93+
94+
### PYTHON / PLATFORM
95+
96+
Shows Python version and OS information for both environments:
97+
98+
```
99+
=== PYTHON / PLATFORM ===
100+
Local:
101+
Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
102+
Impl : CPython
103+
Exec : /home/user/.venv/bin/python
104+
OS : Linux 5.15.0-generic (x86_64)
105+
106+
Remote:
107+
Python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
108+
Impl : CPython
109+
Exec : /home/ray/anaconda3/bin/python
110+
OS : Linux 5.4.0-aws (x86_64)
111+
```
112+
113+
<Warning>
114+
Watch for different Python versions or different OS types (macOS local vs Linux remote).
115+
</Warning>
116+
117+
#### Architecture Mismatch (macOS to Linux)
118+
119+
If you see different OS types (e.g., `Darwin` locally vs `Linux` remotely), compiled extensions may fail with `ModuleNotFoundError` or segfaults.
120+
121+
**Solution**: Run Geneva from the same OS/architecture as your cluster (typically Linux x86_64). Use a Linux VM, container, or remote development environment.
122+
123+
### Environment Variables
124+
125+
Environment variables present in only one environment:
126+
127+
```
128+
=== ENV VARS: keys only in LOCAL ===
129+
+ CONDA_PREFIX
130+
+ VIRTUAL_ENV
131+
132+
=== ENV VARS: keys only in REMOTE ===
133+
+ RAY_ADDRESS
134+
+ KUBERNETES_SERVICE_HOST
135+
```
136+
137+
<Warning>
138+
Missing `AWS_*` or `GOOGLE_APPLICATION_CREDENTIALS` can cause storage authentication failures.
139+
</Warning>
140+
141+
#### Passing Environment Variables to Workers
142+
143+
If critical environment variables are missing on workers, you can pass them via the manifest or cluster configuration.
144+
145+
**Option 1: Via Manifest**
146+
147+
<CodeGroup>
148+
```python Python icon="python"
149+
from geneva.manifest.builder import GenevaManifestBuilder
150+
import os
151+
152+
manifest = (
153+
GenevaManifestBuilder.create("my-manifest")
154+
.env({
155+
"AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
156+
"AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
157+
"MY_API_KEY": os.environ["MY_API_KEY"],
158+
})
159+
.build()
160+
)
161+
```
162+
</CodeGroup>
163+
164+
**Option 2: Via Cluster Configuration**
165+
166+
<CodeGroup>
167+
```python Python icon="python"
168+
from geneva.cluster.builder import GenevaClusterBuilder
169+
import os
170+
171+
cluster = (
172+
GenevaClusterBuilder.create("my-cluster")
173+
.ray_init_kwargs({
174+
"runtime_env": {
175+
"env_vars": {
176+
"AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
177+
"AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
178+
}
179+
}
180+
})
181+
.build()
182+
)
183+
```
184+
</CodeGroup>
185+
186+
<Warning>
187+
Avoid hardcoding secrets. Use `os.environ` to pass values from your local environment, or use a secrets manager in production.
188+
</Warning>
189+
190+
### Packages
191+
192+
The tool shows version mismatches and packages only present in one environment:
193+
194+
```
195+
=== PACKAGES: version mismatches ===
196+
* numpy: local=1.26.4 remote=2.2.6
197+
* torch: local=2.0.1 remote=2.8.0+cpu
198+
* attrs: local=23.2.0 remote=24.2.0
199+
* pyarrow: local=14.0.1 remote=17.0.0
200+
201+
=== PACKAGES: only in LOCAL ===
202+
+ my-custom-package
203+
+ dev-tools
204+
205+
=== PACKAGES: only in REMOTE ===
206+
+ kuberay-client
207+
```
208+
209+
<Warning>
210+
Watch for major version differences (NumPy 1.x vs 2.x) and PyTorch version mismatches.
211+
</Warning>
212+
213+
#### Common Package Issues
214+
215+
| Issue | Symptoms | Fix |
216+
|-------|----------|-----|
217+
| **NumPy 1.x vs 2.x** | `ArrowInvalid`, `ValueError: cannot convert`, serialization failures | Pin `numpy==1.26.4` |
218+
| **PyTorch mismatch** | Model loading failures, CUDA errors, unexpected inference results | Pin to matching `torch` version |
219+
| **attrs mismatch** | `TypeError: Enum.__new__() missing 1 required positional argument` | Pin `attrs` to local version |
220+
| **Missing package** | `ModuleNotFoundError: No module named 'xyz'` | Add package to manifest |
221+
222+
#### Fixing Package Mismatches
223+
224+
**Option 1: Manifest pip Dependencies**
225+
226+
Specify packages in a Geneva manifest for a quick fix:
227+
228+
<CodeGroup>
229+
```python Python icon="python"
230+
from geneva.manifest.builder import GenevaManifestBuilder
231+
232+
manifest = (
233+
GenevaManifestBuilder.create("my-manifest")
234+
.pip([
235+
"numpy==1.26.4",
236+
"torch==2.0.1",
237+
"attrs==23.2.0",
238+
])
239+
.build()
240+
)
241+
242+
# Then use with db.context()
243+
conn = geneva.connect("s3://my-bucket/my-db")
244+
conn.define_manifest("my-manifest", manifest)
245+
with conn.context(cluster="my-cluster", manifest="my-manifest"):
246+
conn.open_table("my-table").backfill("my-column")
247+
```
248+
</CodeGroup>
249+
250+
*Pros*: Quick, reusable across sessions, stored with your database.
251+
252+
*Cons*: Slower startup (downloads packages), may not work for complex dependencies.
253+
254+
**Option 2: Custom Ray Worker Image**
255+
256+
For KubeRay deployments, build a custom worker image:
257+
258+
<CodeGroup>
259+
```dockerfile Dockerfile icon="docker"
260+
# Dockerfile.ray-worker
261+
FROM rayproject/ray:2.30.0-py311
262+
263+
# Install exact versions
264+
RUN pip install \
265+
numpy==1.26.4 \
266+
torch==2.0.1 \
267+
attrs==23.2.0 \
268+
geneva==0.8.0
269+
270+
# Copy any custom packages
271+
COPY ./my_udfs /app/my_udfs
272+
```
273+
</CodeGroup>
274+
275+
Then reference in RayCluster spec:
276+
277+
<CodeGroup>
278+
```yaml Kubernetes icon="kubernetes"
279+
spec:
280+
workerGroupSpecs:
281+
- template:
282+
spec:
283+
containers:
284+
- image: your-registry/ray-worker:latest
285+
```
286+
</CodeGroup>
287+
288+
*Pros*: Fastest startup, reproducible.
289+
290+
*Cons*: Requires image build/push workflow.
291+
292+
**Option 3: Conda Environment**
293+
294+
Use a conda environment on workers via the cluster builder:
295+
296+
<CodeGroup>
297+
```python Python icon="python"
298+
from geneva.cluster.builder import GenevaClusterBuilder
299+
300+
cluster = (
301+
GenevaClusterBuilder.create("my-cluster")
302+
.ray_init_kwargs({
303+
"runtime_env": {"conda": "environment.yml"}
304+
})
305+
.build()
306+
)
307+
```
308+
</CodeGroup>
309+
310+
Or specify conda channels and dependencies inline:
311+
312+
<CodeGroup>
313+
```python Python icon="python"
314+
cluster = (
315+
GenevaClusterBuilder.create("my-cluster")
316+
.ray_init_kwargs({
317+
"runtime_env": {
318+
"conda": {
319+
"channels": ["conda-forge"],
320+
"dependencies": [
321+
"python=3.10",
322+
"ffmpeg<8",
323+
"torchvision=0.22.1"
324+
]
325+
},
326+
"config": {"eager_install": True}
327+
}
328+
})
329+
.build()
330+
)
331+
```
332+
</CodeGroup>
333+
334+
*Pros*: Best for complex dependencies with native libraries (ffmpeg, CUDA).
335+
336+
*Cons*: Slower environment creation, requires conda on workers.
337+
338+
## API Reference
339+
340+
For detailed API documentation on the environment comparison functions, see the [Geneva Diagnostics API Reference](https://lancedb.github.io/geneva/api/diagnostics).

0 commit comments

Comments
 (0)