-
Notifications
You must be signed in to change notification settings - Fork 26
Add nightly dependency bump workflow #479
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 25 commits
e74800c
7252cb3
e56456b
c2d895d
6cc475d
e9c78dd
7349738
ca80bbb
cdbfea2
feb2005
d9d2dfe
e2a6c4c
b074e69
3fb046f
351813e
9d57171
9f0ebbb
60b627a
1544468
a6d58c1
e84deb6
51a9ee0
957024b
076e529
30d7d0a
f3b7a09
5182877
9efd150
5ebbe34
1b56b75
2f26b9c
0329b09
4db8d3e
84706be
d9d3a9a
1357fc8
74c6fb4
f5ba036
8b05265
dda3ac0
63b72f3
1c7bdf3
bb66be4
692abde
12cc0e5
e7fb5f7
15e4550
b1b4a87
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -6,7 +6,14 @@ on: | |||||
- main | ||||||
- "release/v*" | ||||||
- ci-workflow | ||||||
- nightly-dependency-updates | ||||||
workflow_dispatch: # be able to run the workflow on demand | ||||||
workflow_call: | ||||||
inputs: | ||||||
ref: | ||||||
description: 'Git ref to checkout' | ||||||
required: false | ||||||
type: string | ||||||
env: | ||||||
AWS_DEFAULT_REGION: us-east-1 | ||||||
STAGING_ECR_REGISTRY: 637423224110.dkr.ecr.us-east-1.amazonaws.com | ||||||
|
@@ -34,6 +41,8 @@ jobs: | |||||
steps: | ||||||
- name: Checkout Repo @ SHA - ${{ github.sha }} | ||||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 #5.0.0 | ||||||
with: | ||||||
ref: ${{ inputs.ref || github.sha }} | ||||||
Comment on lines
+43
to
+44
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why |
||||||
|
||||||
- name: Get Python Distro Output | ||||||
id: python_output | ||||||
|
@@ -114,7 +123,7 @@ jobs: | |||||
name: "Publish Main Build Status" | ||||||
needs: [ build, application-signals-e2e-test ] | ||||||
runs-on: ubuntu-latest | ||||||
if: always() | ||||||
if: always() && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/release/')) | ||||||
|
if: always() && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/release/')) | |
if: (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/heads/release/v')) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How did we confirm that refs/heads/main
is correct?
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the workflow should be:
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
name: Nightly Upstream Snapshot Build | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Lets add a description |
||
|
||
on: | ||
schedule: | ||
- cron: "21 3 * * *" | ||
workflow_dispatch: | ||
|
||
env: | ||
BRANCH_NAME: nightly-dependency-updates | ||
|
||
jobs: | ||
update-and-create-pr: | ||
runs-on: ubuntu-latest | ||
outputs: | ||
has_changes: ${{ steps.check_changes.outputs.has_changes }} | ||
otel_python_version: ${{ steps.get_versions.outputs.otel_python_version }} | ||
otel_contrib_version: ${{ steps.get_versions.outputs.otel_contrib_version }} | ||
|
||
steps: | ||
- name: Checkout repository | ||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 #5.0.0 | ||
with: | ||
fetch-depth: 0 | ||
token: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Check if nightly branch already exists | ||
thpierce marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
run: | | ||
if git ls-remote --exit-code --heads origin "$BRANCH_NAME"; then | ||
echo "Branch $BRANCH_NAME already exists. Skipping run to avoid conflicts." | ||
echo "Please merge or close the existing PR before the next nightly run." | ||
exit 1 | ||
fi | ||
- name: Configure git and create branch | ||
run: | | ||
git config --local user.email "[email protected]" | ||
git config --local user.name "GitHub Action" | ||
git checkout -b "$BRANCH_NAME" | ||
- name: Set up Python | ||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c #v6.0.0 | ||
with: | ||
python-version: '3.11' | ||
|
||
|
||
- name: Install build tools | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install toml requests | ||
- name: Get latest upstream versions | ||
id: get_versions | ||
run: python scripts/get_upstream_versions.py | ||
|
||
- name: Update dependencies | ||
env: | ||
OTEL_PYTHON_VERSION: ${{ steps.get_versions.outputs.otel_python_version }} | ||
OTEL_CONTRIB_VERSION: ${{ steps.get_versions.outputs.otel_contrib_version }} | ||
run: python scripts/update_dependencies.py | ||
|
||
- name: Check for changes and create PR | ||
id: check_changes | ||
run: | | ||
if git diff --quiet; then | ||
echo "No dependency updates needed" | ||
echo "has_changes=false" >> $GITHUB_OUTPUT | ||
else | ||
echo "Dependencies were updated" | ||
echo "has_changes=true" >> $GITHUB_OUTPUT | ||
# Create PR | ||
git add aws-opentelemetry-distro/pyproject.toml | ||
|
||
git commit -m "chore: update OpenTelemetry dependencies to ${{ steps.get_versions.outputs.otel_python_version }}/${{ steps.get_versions.outputs.otel_contrib_version }}" | ||
git push origin "$BRANCH_NAME" | ||
gh pr create \ | ||
--title "Nightly dependency update: OpenTelemetry ${{ steps.get_versions.outputs.otel_python_version }}/${{ steps.get_versions.outputs.otel_contrib_version }}" \ | ||
--body "Automated update of OpenTelemetry dependencies. | ||
**Updated versions:** | ||
- OpenTelemetry Python: ${{ steps.get_versions.outputs.otel_python_version }} | ||
- OpenTelemetry Contrib: ${{ steps.get_versions.outputs.otel_contrib_version }}" \ | ||
--base main \ | ||
--head "$BRANCH_NAME" | ||
fi | ||
env: | ||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
build-and-test: | ||
needs: update-and-create-pr | ||
if: needs.update-and-create-pr.outputs.has_changes == 'true' | ||
uses: ./.github/workflows/main-build.yml | ||
secrets: inherit | ||
permissions: | ||
id-token: write | ||
contents: read | ||
with: | ||
ref: nightly-dependency-updates |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
#!/usr/bin/env python3 | ||
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
import os | ||
import re | ||
import sys | ||
|
||
import requests | ||
|
||
|
||
def get_latest_otel_versions(): | ||
"""Get latest OpenTelemetry versions from GitHub releases.""" | ||
try: | ||
# Query GitHub API for latest release | ||
response = requests.get( | ||
"https://api.github.com/repos/open-telemetry/opentelemetry-python/releases/latest", timeout=30 | ||
) | ||
response.raise_for_status() | ||
|
||
release_data = response.json() | ||
release_title = release_data["name"] | ||
|
||
# Parse "Version 1.37.0/0.58b0" format | ||
match = re.search(r"Version\s+(\d+\.\d+\.\d+)/(\d+\.\d+b\d+)", release_title) | ||
Comment on lines
+24
to
+25
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Callout that this format has changed in past (look at old releases). It's ok for this to be a bit fragile for now, but maybe a better way would be to use pip or something and just check current |
||
if not match: | ||
print(f"Could not parse release title: {release_title}") | ||
sys.exit(1) | ||
|
||
otel_python_version = match.group(1) | ||
otel_contrib_version = match.group(2) | ||
thpierce marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
return otel_python_version, otel_contrib_version | ||
|
||
except requests.RequestException as request_error: | ||
print(f"Error getting OpenTelemetry versions: {request_error}") | ||
sys.exit(1) | ||
|
||
|
||
def main(): | ||
otel_python_version, otel_contrib_version = get_latest_otel_versions() | ||
|
||
print(f"OTEL_PYTHON_VERSION={otel_python_version}") | ||
print(f"OTEL_CONTRIB_VERSION={otel_contrib_version}") | ||
|
||
# Write to GitHub output if in CI | ||
if "GITHUB_OUTPUT" in os.environ: | ||
with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as output_file: | ||
output_file.write(f"otel_python_version={otel_python_version}\n") | ||
output_file.write(f"otel_contrib_version={otel_contrib_version}\n") | ||
|
||
|
||
if __name__ == "__main__": | ||
main() |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,149 @@ | ||
#!/usr/bin/env python3 | ||
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
# SPDX-License-Identifier: Apache-2.0 | ||
|
||
import os | ||
import re | ||
import sys | ||
|
||
import requests | ||
|
||
# Dependencies that use the first version number (opentelemetry-python) | ||
thpierce marked this conversation as resolved.
Show resolved
Hide resolved
|
||
PYTHON_CORE_DEPS = [ | ||
"opentelemetry-api", | ||
"opentelemetry-sdk", | ||
"opentelemetry-exporter-otlp", | ||
"opentelemetry-exporter-otlp-proto-grpc", | ||
"opentelemetry-exporter-otlp-proto-http", | ||
"opentelemetry-propagator-b3", | ||
"opentelemetry-propagator-jaeger", | ||
"opentelemetry-exporter-otlp-proto-common", | ||
] | ||
|
||
# Dependencies that use the second version number (opentelemetry-python-contrib) | ||
CONTRIB_DEPS = [ | ||
"opentelemetry-distro", | ||
"opentelemetry-processor-baggage", | ||
"opentelemetry-propagator-ot-trace", | ||
"opentelemetry-instrumentation", | ||
"opentelemetry-instrumentation-aws-lambda", | ||
"opentelemetry-instrumentation-aio-pika", | ||
"opentelemetry-instrumentation-aiohttp-client", | ||
"opentelemetry-instrumentation-aiopg", | ||
"opentelemetry-instrumentation-asgi", | ||
"opentelemetry-instrumentation-asyncpg", | ||
"opentelemetry-instrumentation-boto", | ||
"opentelemetry-instrumentation-boto3sqs", | ||
"opentelemetry-instrumentation-botocore", | ||
"opentelemetry-instrumentation-celery", | ||
"opentelemetry-instrumentation-confluent-kafka", | ||
"opentelemetry-instrumentation-dbapi", | ||
"opentelemetry-instrumentation-django", | ||
"opentelemetry-instrumentation-elasticsearch", | ||
"opentelemetry-instrumentation-falcon", | ||
"opentelemetry-instrumentation-fastapi", | ||
"opentelemetry-instrumentation-flask", | ||
"opentelemetry-instrumentation-grpc", | ||
"opentelemetry-instrumentation-httpx", | ||
"opentelemetry-instrumentation-jinja2", | ||
"opentelemetry-instrumentation-kafka-python", | ||
"opentelemetry-instrumentation-logging", | ||
"opentelemetry-instrumentation-mysql", | ||
"opentelemetry-instrumentation-mysqlclient", | ||
"opentelemetry-instrumentation-pika", | ||
"opentelemetry-instrumentation-psycopg2", | ||
"opentelemetry-instrumentation-pymemcache", | ||
"opentelemetry-instrumentation-pymongo", | ||
"opentelemetry-instrumentation-pymysql", | ||
"opentelemetry-instrumentation-pyramid", | ||
"opentelemetry-instrumentation-redis", | ||
"opentelemetry-instrumentation-remoulade", | ||
"opentelemetry-instrumentation-requests", | ||
"opentelemetry-instrumentation-sqlalchemy", | ||
"opentelemetry-instrumentation-sqlite3", | ||
"opentelemetry-instrumentation-starlette", | ||
"opentelemetry-instrumentation-system-metrics", | ||
"opentelemetry-instrumentation-tornado", | ||
"opentelemetry-instrumentation-tortoiseorm", | ||
"opentelemetry-instrumentation-urllib", | ||
"opentelemetry-instrumentation-urllib3", | ||
"opentelemetry-instrumentation-wsgi", | ||
"opentelemetry-instrumentation-cassandra", | ||
] | ||
|
||
# AWS-specific packages with independent versioning | ||
AWS_DEPS = [ | ||
"opentelemetry-sdk-extension-aws", | ||
"opentelemetry-propagator-aws-xray", | ||
] | ||
|
||
|
||
def get_latest_version(package_name): | ||
thpierce marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IMO we should put this into |
||
"""Get the latest version of a package from PyPI.""" | ||
try: | ||
response = requests.get(f"https://pypi.org/pypi/{package_name}/json", timeout=30) | ||
response.raise_for_status() | ||
data = response.json() | ||
return data["info"]["version"] | ||
except requests.RequestException as request_error: | ||
print(f"Warning: Could not get latest version for {package_name}: {request_error}") | ||
return None | ||
Comment on lines
+82
to
+89
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IMO There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we hit an exception, just fail the job, I don't want a partial update really. Maybe add a retry or something. |
||
|
||
|
||
def main(): | ||
otel_python_version = os.environ.get("OTEL_PYTHON_VERSION") | ||
otel_contrib_version = os.environ.get("OTEL_CONTRIB_VERSION") | ||
|
||
if not otel_python_version or not otel_contrib_version: | ||
print("Error: OTEL_PYTHON_VERSION and OTEL_CONTRIB_VERSION environment variables required") | ||
sys.exit(1) | ||
|
||
pyproject_path = "aws-opentelemetry-distro/pyproject.toml" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually there are a few files to change: https://github.com/search?q=repo%3Aaws-observability%2Faws-otel-python-instrumentation+%2Fopentelemetry-.*%3D%3D%2F&type=code Though I think we may be able to remove |
||
|
||
try: | ||
with open(pyproject_path, "r", encoding="utf-8") as input_file: | ||
content = input_file.read() | ||
|
||
updated = False | ||
|
||
# Update opentelemetry-python dependencies | ||
for dep in PYTHON_CORE_DEPS: | ||
pattern = rf'"{re.escape(dep)} == [^"]*"' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What does |
||
replacement = f'"{dep} == {otel_python_version}"' | ||
if re.search(pattern, content): | ||
content = re.sub(pattern, replacement, content) | ||
updated = True | ||
|
||
# Update opentelemetry-python-contrib dependencies | ||
for dep in CONTRIB_DEPS: | ||
pattern = rf'"{re.escape(dep)} == [^"]*"' | ||
replacement = f'"{dep} == {otel_contrib_version}"' | ||
if re.search(pattern, content): | ||
content = re.sub(pattern, replacement, content) | ||
updated = True | ||
|
||
# Update dependencies with independent versioning | ||
for dep in AWS_DEPS: | ||
latest_version = get_latest_version(dep) | ||
if latest_version: | ||
pattern = rf'"{re.escape(dep)} == [^"]*"' | ||
replacement = f'"{dep} == {latest_version}"' | ||
if re.search(pattern, content): | ||
content = re.sub(pattern, replacement, content) | ||
updated = True | ||
print(f"Updated {dep} to {latest_version}") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why print? |
||
|
||
if updated: | ||
with open(pyproject_path, "w", encoding="utf-8") as output_file: | ||
output_file.write(content) | ||
print(f"Dependencies updated to Python {otel_python_version} / Contrib {otel_contrib_version}") | ||
else: | ||
print("No OpenTelemetry dependencies found to update") | ||
|
||
except (OSError, IOError) as file_error: | ||
print(f"Error updating dependencies: {file_error}") | ||
sys.exit(1) | ||
|
||
|
||
if __name__ == "__main__": | ||
main() |
Uh oh!
There was an error while loading. Please reload this page.