Skip to content

Commit 14ec182

Browse files
leandrodamascenaleandrodamascena
authored
docs(build_recipes): add cross build page (#7199)
* Adding cross build page * Adding cross build page * Adding cross build page
1 parent ae8fdb2 commit 14ec182

File tree

11 files changed

+426
-0
lines changed

11 files changed

+426
-0
lines changed

‎docs/build_recipes/cross-platform.md

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
---
2+
title: Cross-Platform Build Considerations
3+
description: Handle architecture differences when building AWS Lambda packages
4+
---
5+
6+
<!-- markdownlint-disable MD043 -->
7+
8+
Many modern Python packages include compiled extensions written in Rust or C/C++ for performance reasons. These compiled components are platform-specific and can cause deployment issues when building on different architectures.
9+
10+
???+ warning "Architecture Mismatch Issues"
11+
Building AWS Lambda packages on macOS (ARM64/Intel) for deployment on AWS Lambda (Linux x86_64 or ARM64) will result in incompatible binary dependencies that cause import errors at runtime.
12+
13+
## Common compiled libraries
14+
15+
Taking into consideration Powertools for AWS dependencies and common Python packages, these libraries include compiled Rust/C components that require architecture-specific builds:
16+
17+
| Library | Language | Components | Impact | Used in Powertools for AWS|
18+
|---------|----------|------------|--------|-------------------|
19+
| **pydantic** | Rust | Core validation engine | High - Core functionality affected | ✅ Core dependency |
20+
| **aws-encryption-sdk** | C | Encryption/decryption | High - Data masking fails | ✅ Optional (datamasking extra) |
21+
| **protobuf** | C++ | Protocol buffer serialization | High - Message parsing fails | ✅ Optional (kafka-consumer-protobuf) |
22+
| **redis** | C | Redis client with hiredis | Medium - Falls back to pure Python | ✅ Optional (redis extra) |
23+
| **valkey-glide** | Rust | High-performance Redis client | High - Client completely broken | ✅ Optional (valkey extra) |
24+
| **orjson** | Rust | JSON serialization | Medium - Performance degradation | ❌ Not used (but common) |
25+
| **uvloop** | C | Event loop implementation | Medium - Falls back to asyncio | ❌ Not used (but common) |
26+
| **lxml** | C | XML/HTML processing | High - XML parsing fails | ❌ Not used (but common) |
27+
28+
## Powertools extras dependencies and architecture
29+
30+
Different Powertools for AWS extras dependencies have varying levels of architecture dependency:
31+
32+
=== "Safe extras (pure python)"
33+
34+
```txt title="requirements.txt - Safe for any platform"
35+
# The 'all' extra includes both safe and architecture-dependent packages
36+
aws-lambda-powertools[all]==3.18.0
37+
38+
# This is equivalent to:
39+
# pydantic, pydantic-settings, aws-xray-sdk, fastjsonschema,
40+
# aws-encryption-sdk, jsonpath-ng
41+
```
42+
43+
=== "Architecture-dependent extras"
44+
45+
```txt title="requirements.txt - Requires Linux builds"
46+
# These extras include compiled dependencies
47+
aws-lambda-powertools[parser]==3.18.0 # pydantic (Rust)
48+
aws-lambda-powertools[validation]==3.18.0 # fastjsonschema (C)
49+
aws-lambda-powertools[datamasking]==3.18.0 # aws-encryption-sdk (C)
50+
aws-lambda-powertools[redis]==3.18.0 # redis with hiredis (C)
51+
aws-lambda-powertools[valkey]==3.18.0 # valkey-glide (Rust)
52+
aws-lambda-powertools[kafka-consumer-avro]==3.18.0 # avro (C)
53+
aws-lambda-powertools[kafka-consumer-protobuf]==3.18.0 # protobuf (C++)
54+
```
55+
56+
=== "All extras (mixed dependencies)"
57+
58+
```txt title="requirements.txt - Requires careful platform handling"
59+
# These extras have minimal or no compiled dependencies
60+
aws-lambda-powertools[tracer]==3.18.0 # aws-xray-sdk (mostly pure Python)
61+
aws-lambda-powertools[aws-sdk]==3.18.0 # boto3 (pure Python)
62+
63+
```
64+
65+
???+ tip "Powertools for AWS build strategy"
66+
1. **Use `[all]` extra with Docker builds** for maximum compatibility
67+
2. **Use specific extras** if you want to avoid certain compiled dependencies
68+
3. **Test imports** after building to catch architecture mismatches early
69+
70+
## Understanding Python wheels
71+
72+
Python wheels are binary distribution packages that include compiled extensions. Lambda requires specific wheel types based on the target runtime architecture and GLIBC version.
73+
74+
### Wheel naming convention
75+
76+
Wheels follow a specific naming pattern that indicates compatibility:
77+
78+
```txt
79+
{package}-{version}-{python tag}-{abi tag}-{platform tag}.whl
80+
```
81+
82+
**Example breakdown:**
83+
84+
```txt
85+
pydantic-2.5.0-cp311-cp311-linux_x86_64.whl
86+
│ │ │ │ └─ Platform: Linux x86_64
87+
│ │ │ └─ ABI: CPython 3.11
88+
│ │ └─ Python: CPython 3.11
89+
│ └─ Version: 2.5.0
90+
└─ Package: pydantic
91+
```
92+
93+
### Platform tags and Lambda compatibility
94+
95+
| Platform Tag | Description | Lambda Compatibility |
96+
|--------------|-------------|---------------------|
97+
| `linux_x86_64` | Linux 64-bit Intel/AMD | ✅ Lambda x86_64 |
98+
| `linux_aarch64` | Linux 64-bit ARM | ✅ Lambda arm64 |
99+
| `macosx_*` | macOS (any version) | ❌ Not compatible |
100+
| `win_*` | Windows (any version) | ❌ Not compatible |
101+
| `any` | Pure Python, no compiled code | ✅ All platforms |
102+
103+
### Source distributions vs wheels
104+
105+
| Package Type | Extension | Compilation | Lambda Recommendation |
106+
|--------------|-----------|-------------|----------------------|
107+
| **Wheel** | `.whl` | Pre-compiled | ✅ Preferred - faster, consistent |
108+
| **Source Distribution** | `.tar.gz` | Compile during install | ❌ Avoid - platform-dependent compilation |
109+
110+
**Why wheels matter for Lambda:**
111+
112+
* **Consistent builds** - Same binary across environments
113+
* **Faster installs** - No compilation step required
114+
* **Predictable dependencies** - Known system library requirements
115+
* **Architecture safety** - Platform-specific binaries
116+
117+
## Lambda runtime environments
118+
119+
<!-- markdownlint-disable MD013 -->
120+
Lambda managed runtimes use [specific Amazon Linux versions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html#runtimes-supported) versions with fixed GLIBC versions. Packages built on development machines with newer GLIBC versions will fail at runtime with import errors. Each Python runtime version corresponds to a specific Amazon Linux base system that determines compatible system library versions.
121+
<!-- markdownlint-enable MD013 -->
122+
123+
### Amazon Linux versions and GLIBC compatibility
124+
125+
| Python Runtime | Base System | GLIBC Version | Architecture Support |
126+
|----------------|-------------------|---------------|----------------------|
127+
| **python3.9** | Amazon Linux 2 | 2.26 | x86_64, arm64 |
128+
| **python3.10** | Amazon Linux 2 | 2.26 | x86_64, arm64 |
129+
| **python3.11** | Amazon Linux 2 | 2.26 | x86_64, arm64 |
130+
| **python3.12** | Amazon Linux 2023 | 2.34 | x86_64, arm64 |
131+
| **python3.13** | Amazon Linux 2023 | 2.34 | x86_64, arm64 |
132+
133+
???+ warning "GLIBC Version Mismatch"
134+
Compiled libraries built on systems with newer GLIBC versions will fail on Lambda runtimes with older GLIBC versions. Ubuntu 24.04 (GLIBC 2.39) and Ubuntu 22.04 (GLIBC 2.35) are incompatible with Lambda python3.11 and earlier (GLIBC 2.26). Always use `--platform` flags or Docker with Lambda base images.
135+
136+
### Manylinux compatibility tags
137+
138+
Python wheels use manylinux tags to indicate GLIBC compatibility. Understanding these tags helps you choose the right wheels for Lambda:
139+
140+
| manylinux Tag | GLIBC Version | Lambda Compatibility | Recommendation |
141+
|---------------|---------------|---------------------|----------------|
142+
| **manylinux1** | 2.5 | ✅ All runtimes | Legacy, limited package support |
143+
| **manylinux2010** | 2.12 | ✅ All runtimes | Good compatibility |
144+
| **manylinux2014** | 2.17 | ✅ All runtimes | Recommended for most packages |
145+
| **manylinux_2_17** | 2.17 | ✅ All runtimes | Modern standard |
146+
| **manylinux_2_24** | 2.24 | ✅ All runtimes | Good for newer packages |
147+
| **manylinux_2_28** | 2.28 | ✅ python3.12+, ❌ python3.11- | Use with caution |
148+
| **manylinux_2_34** | 2.34 | ✅ python3.12+, ❌ python3.11- | AL2023 only |
149+
150+
## Runtime-specific considerations
151+
152+
### Amazon Linux 2 (python3.8 - python3.11)
153+
154+
Amazon Linux 2 is based on RHEL 7 and uses an older GLIBC version (2.26). This provides broad compatibility but may limit access to newer compiled features.
155+
156+
**Characteristics:**
157+
158+
* **GLIBC 2.26** - Compatible with most manylinux wheels
159+
* **OpenSSL 1.0.2** - Legacy TLS support
160+
161+
**Best practices:**
162+
163+
```bash
164+
--8<-- "examples/build_recipes/build_multi_arch/build-al2.sh"
165+
```
166+
167+
### Amazon Linux 2023 (python3.12+)
168+
169+
Amazon Linux 2023 is a modern, minimal Linux distribution with updated system libraries and better security.
170+
171+
**Characteristics:**
172+
173+
* **GLIBC 2.34** - Supports newer compiled libraries
174+
* **OpenSSL 3.0** - Latest TLS and cryptographic features
175+
* **Smaller footprint** - Optimized for containers and serverless
176+
177+
**Migration considerations:**
178+
179+
```bash
180+
--8<-- "examples/build_recipes/build_multi_arch/build-al2023.sh"
181+
```
182+
183+
## Multi-platform build strategies
184+
185+
=== "Docker-based Builds (Recommended)"
186+
187+
Use AWS Lambda base images to ensure Linux x86_64 or ARM64 compatibility:
188+
189+
=== "Dockerfile"
190+
191+
```dockerfile
192+
--8<-- "examples/build_recipes/build_multi_arch/Dockerfile.lambda"
193+
```
194+
195+
=== "Build Script"
196+
197+
```bash
198+
--8<-- "examples/build_recipes/build_multi_arch/build-multiplatform.sh"
199+
```
200+
201+
=== "Platform-specific pip install"
202+
203+
Force installation of Linux-compatible wheels:
204+
205+
=== "Build Script"
206+
207+
```bash
208+
--8<-- "examples/build_recipes/build_multi_arch/build-linux-wheels.sh"
209+
```
210+
211+
=== "GitHub Actions multi-arch"
212+
213+
Use GitHub Actions with Linux runners for consistent builds:
214+
215+
=== "Workflow"
216+
217+
```yaml
218+
--8<-- "examples/build_recipes/build_multi_arch/lambda-build.yml"
219+
```
220+
221+
## Debugging compatibility issues
222+
223+
When you encounter runtime errors related to compiled dependencies, use these techniques to diagnose and fix the issues:
224+
225+
### Common error patterns
226+
227+
=== "GLIBC version errors"
228+
229+
```bash
230+
--8<-- "examples/build_recipes/build_multi_arch/debug-glibc.sh"
231+
```
232+
233+
=== "Architecture mismatch"
234+
235+
```bash
236+
--8<-- "examples/build_recipes/build_multi_arch/debug-arch-mismatch.sh"
237+
```
238+
239+
=== "Missing system libraries"
240+
241+
```bash
242+
--8<-- "examples/build_recipes/build_multi_arch/debug-missing-libs.sh"
243+
```
244+
245+
## Best practices for cross-platform builds
246+
247+
???+ tip "Development Workflow"
248+
Develop locally on your preferred platform, but always build deployment packages in a Linux environment or Docker container to ensure compatibility.
249+
250+
1. **Always build on Linux** for Lambda deployments, or use Docker with Lambda base images
251+
2. **Use `--platform` flags** when installing with pip to force Linux-compatible wheels
252+
3. **Test imports** in your build environment before deployment
253+
4. **Pin dependency versions** to ensure reproducible builds across platforms
254+
5. **Use CI/CD with Linux runners** to avoid local architecture issues
255+
6. **Consider Lambda container images** for complex dependency scenarios

‎examples/build_recipes/build_multi_arch/Dockerfile.lambda

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#Public Lambda image
2+
FROM public.ecr.aws/lambda/python@sha256:7e7f098baa11a527fbe59f33f4ed032a36b6e87b22ea73da1175522095885f74
3+
4+
# Set workdir file
5+
WORKDIR /tmp/app
6+
7+
# Copy requirements first for better caching
8+
COPY requirements.txt .
9+
10+
# Install dependencies in Lambda-compatible environment
11+
RUN pip install -r requirements.txt
12+
13+
# Copy application code
14+
COPY . .
15+
16+
CMD ["app.lambda_handler"]

‎examples/build_recipes/build_multi_arch/build-al2.sh

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/bash
2+
# Use Amazon Linux 2 base image for builds
3+
docker run --rm -v "$PWD":/var/task \
4+
public.ecr.aws/lambda/python:3.11 \
5+
pip install -r requirements.txt -t /var/task/

‎examples/build_recipes/build_multi_arch/build-al2023.sh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash
2+
# Some packages may require rebuilding for AL2023
3+
# Check for GLIBC symbol errors in logs:
4+
# ImportError: /lib64/libc.so.6: version `GLIBC_2.34' not found
5+
6+
# Use AL2023 base image for python3.12+
7+
docker run --rm -v "$PWD":/var/task \
8+
public.ecr.aws/lambda/python:3.12 \
9+
pip install -r requirements.txt -t /var/task/

‎examples/build_recipes/build_multi_arch/build-linux-wheels.sh

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
3+
# Create build directory
4+
mkdir -p build/
5+
6+
# Install Linux-compatible wheels
7+
pip install \
8+
--platform manylinux2014_x86_64 \
9+
--target build/ \
10+
--implementation cp \
11+
--python-version 3.13 \
12+
--only-binary=:all: \
13+
--upgrade \
14+
--abi cp313 \
15+
-r requirements.txt
16+
17+
# Copy application code
18+
cp -r src/* build/
19+
20+
# Create deployment package
21+
cd build && zip -r ../lambda-linux.zip . && cd ..
22+
23+
echo "✅ Linux-compatible package created"

‎examples/build_recipes/build_multi_arch/build-multiplatform.sh

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/bash
2+
3+
# Build using Lambda-compatible environment
4+
docker build -f Dockerfile.lambda -t lambda-build .
5+
6+
# Extract built packages
7+
docker create --name temp-container lambda-build
8+
docker cp temp-container:/var/task ./build
9+
docker rm temp-container
10+
11+
# Create deployment package
12+
cd build && zip -r ../lambda-multiplatform.zip . && cd ..
13+
14+
echo "✅ Multi-platform compatible package created"

‎examples/build_recipes/build_multi_arch/debug-arch-mismatch.sh

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/bin/bash
2+
# Error message:
3+
# ImportError: cannot import name '_speedups' from 'pydantic'
4+
5+
# Check library architecture
6+
file /opt/python/lib/python3.11/site-packages/pydantic/_internal/_pydantic_core.so
7+
8+
# Expected output for Lambda x86_64:
9+
# ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked
10+
11+
# Solution: Force correct platform
12+
pip install --platform manylinux2014_x86_64 --force-reinstall pydantic -t build/

‎examples/build_recipes/build_multi_arch/debug-glibc.sh

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/bin/bash
2+
# Error message:
3+
# ImportError: /lib64/libc.so.6: version `GLIBC_2.34' not found
4+
5+
# Check GLIBC version in Lambda runtime
6+
ldd --version
7+
8+
# Check required GLIBC symbols in a library
9+
objdump -T /opt/python/lib/python3.11/site-packages/pydantic/_internal/_pydantic_core.so | grep GLIBC
10+
11+
# Solution: Rebuild with compatible base image
12+
docker run --rm -v "$PWD":/var/task public.ecr.aws/lambda/python:3.11 \
13+
pip install --force-reinstall pydantic -t /var/task/

‎examples/build_recipes/build_multi_arch/debug-missing-libs.sh

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash
2+
# Error message:
3+
# ImportError: libffi.so.6: cannot open shared object file
4+
5+
# Check library dependencies
6+
ldd /opt/python/lib/python3.11/site-packages/some_package/_extension.so
7+
8+
# Solution: Use Lambda base image with system dependencies
9+
# Or switch to pure Python alternatives

0 commit comments

Comments
 (0)