-
Notifications
You must be signed in to change notification settings - Fork 32
🐛✨ Opentelemetry instrument asyncpg for aiohttp servers and introduce decorator to generate opentelemtry profile span
#8070
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
bisgaard-itis
merged 32 commits into
ITISFoundation:master
from
bisgaard-itis:8065-instrument-asyncpg
Jul 10, 2025
Merged
Changes from all commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
e44ac88
add opentelemetry-instrumentation-asyncpg dependency repo-wide
bisgaard-itis 934fdde
add asyncpg instrumention in aiohttp case
bisgaard-itis eeb2b6e
clean up tracing tests
bisgaard-itis 84b26fa
minor cleanup
bisgaard-itis fe380c6
add decorator to generate pyinstrument profile span
bisgaard-itis 388728d
ensure entire profile is shown
bisgaard-itis 08dbf7b
minor simplification
bisgaard-itis a5dbadc
experiment a bit with new function decorator
bisgaard-itis a8931d2
add test for handler decorator
bisgaard-itis 01920eb
parametrize test to cover more cases
bisgaard-itis acb7a99
cleanup
bisgaard-itis 51f810a
Merge branch 'master' into 8065-instrument-asyncpg
bisgaard-itis 36718e6
remove comment
bisgaard-itis 31546e4
remove decorator again
bisgaard-itis 0560b2d
Merge branch 'master' into 8065-instrument-asyncpg
bisgaard-itis c464840
enable tracing on local deployment
bisgaard-itis acd8c13
add profile spans for debugging together with @pcrespov
bisgaard-itis 0df115d
fix types
bisgaard-itis f6f82d4
Merge branch 'master' into 8065-instrument-asyncpg
bisgaard-itis 9b41e2f
remove profile spans to see if they are causing tests to fail
bisgaard-itis 48ff90a
Merge branch 'master' into 8065-instrument-asyncpg
bisgaard-itis 78f2200
add tracing when testing storage
bisgaard-itis b128fe2
improve fixtures to mock tracing
bisgaard-itis 44e7632
readd profile spans
bisgaard-itis d0e5b63
Merge branch 'master' into 8065-instrument-asyncpg
mrnicegyu11 121313f
Revert "readd profile spans"
bisgaard-itis 99da31d
Merge branch '8065-instrument-asyncpg' of github.com:bisgaard-itis/os…
bisgaard-itis a3b8d6b
add profile span for @matusdrobuliak66
bisgaard-itis 4197d4c
Merge branch 'master' into 8065-instrument-asyncpg
bisgaard-itis 5650206
Merge branch 'master' into 8065-instrument-asyncpg
bisgaard-itis fbc0aee
add profile span to permissions endpoint
bisgaard-itis c04e853
Merge branch 'master' into 8065-instrument-asyncpg
bisgaard-itis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,12 +17,12 @@ AGENT_VOLUMES_CLEANUP_S3_ENDPOINT=http://172.17.0.1:9001 | |
| AGENT_VOLUMES_CLEANUP_S3_PROVIDER=MINIO | ||
| AGENT_VOLUMES_CLEANUP_S3_REGION=us-east-1 | ||
| AGENT_VOLUMES_CLEANUP_S3_SECRET_KEY=12345678 | ||
| AGENT_TRACING=null | ||
| AGENT_TRACING={} | ||
|
|
||
| API_SERVER_DEV_FEATURES_ENABLED=0 | ||
| API_SERVER_LOGLEVEL=INFO | ||
| API_SERVER_PROFILING=1 | ||
| API_SERVER_TRACING=null | ||
| API_SERVER_TRACING={} | ||
| TRAEFIK_API_SERVER_INFLIGHTREQ_AMOUNT=25 | ||
|
|
||
| AUTOSCALING_DASK=null | ||
|
|
@@ -35,7 +35,7 @@ AUTOSCALING_LOGLEVEL=INFO | |
| AUTOSCALING_NODES_MONITORING=null | ||
| AUTOSCALING_POLL_INTERVAL="00:00:10" | ||
| AUTOSCALING_SSM_ACCESS=null | ||
| AUTOSCALING_TRACING=null | ||
| AUTOSCALING_TRACING={} | ||
|
|
||
| AWS_S3_CLI_S3=null | ||
|
|
||
|
|
@@ -47,7 +47,7 @@ CATALOG_PORT=8000 | |
| CATALOG_PROFILING=1 | ||
| CATALOG_SERVICES_DEFAULT_RESOURCES='{"CPU": {"limit": 0.1, "reservation": 0.1}, "RAM": {"limit": 2147483648, "reservation": 2147483648}}' | ||
| CATALOG_SERVICES_DEFAULT_SPECIFICATIONS='{}' | ||
| CATALOG_TRACING=null | ||
| CATALOG_TRACING={} | ||
|
|
||
| CELERY_RESULT_EXPIRES=P7D | ||
|
|
||
|
|
@@ -63,7 +63,7 @@ CLUSTERS_KEEPER_MAX_MISSED_HEARTBEATS_BEFORE_CLUSTER_TERMINATION=5 | |
| CLUSTERS_KEEPER_PRIMARY_EC2_INSTANCES=null | ||
| CLUSTERS_KEEPER_TASK_INTERVAL=00:00:30 | ||
| CLUSTERS_KEEPER_WORKERS_EC2_INSTANCES=null | ||
| CLUSTERS_KEEPER_TRACING=null | ||
| CLUSTERS_KEEPER_TRACING={} | ||
|
|
||
| DASK_SCHEDULER_HOST=dask-scheduler | ||
| DASK_SCHEDULER_PORT=8786 | ||
|
|
@@ -83,7 +83,7 @@ DIRECTOR_PUBLISHED_HOST_NAME="127.0.0.1:9081" | |
| DIRECTOR_REGISTRY_CACHING_TTL=00:15:00 | ||
| DIRECTOR_REGISTRY_CACHING=True | ||
| DIRECTOR_SERVICES_CUSTOM_CONSTRAINTS=null | ||
| DIRECTOR_TRACING=null | ||
| DIRECTOR_TRACING={} | ||
|
|
||
| DOCKER_API_PROXY_HOST=docker-api-proxy | ||
| DOCKER_API_PROXY_PASSWORD=admin | ||
|
|
@@ -98,11 +98,11 @@ EFS_GROUP_NAME=efs-group | |
| EFS_DNS_NAME=fs-xxx.efs.us-east-1.amazonaws.com | ||
| EFS_MOUNTED_PATH=/tmp/efs | ||
| EFS_PROJECT_SPECIFIC_DATA_DIRECTORY=project-specific-data | ||
| EFS_GUARDIAN_TRACING=null | ||
| EFS_GUARDIAN_TRACING={} | ||
| EFS_DEFAULT_USER_SERVICE_SIZE_BYTES=10000 | ||
|
|
||
| # DATCORE_ADAPTER | ||
| DATCORE_ADAPTER_TRACING=null | ||
| DATCORE_ADAPTER_TRACING={} | ||
|
|
||
| # DIRECTOR_V2 ---- | ||
| COMPUTATIONAL_BACKEND_DEFAULT_CLUSTER_AUTH='{"type":"tls","tls_ca_file":"/home/scu/.dask/dask-crt.pem","tls_client_cert":"/home/scu/.dask/dask-crt.pem","tls_client_key":"/home/scu/.dask/dask-key.pem"}' | ||
|
|
@@ -128,14 +128,14 @@ DYNAMIC_SIDECAR_LOG_LEVEL=DEBUG | |
| DYNAMIC_SIDECAR_PROMETHEUS_MONITORING_NETWORKS=[] | ||
| DYNAMIC_SIDECAR_PROMETHEUS_SERVICE_LABELS={} | ||
| DYNAMIC_SIDECAR_API_SAVE_RESTORE_STATE_TIMEOUT=01:00:00 | ||
| DIRECTOR_V2_TRACING=null | ||
| DIRECTOR_V2_TRACING={} | ||
|
|
||
| # DYNAMIC_SCHEDULER ---- | ||
| DYNAMIC_SCHEDULER_LOGLEVEL=INFO | ||
| DYNAMIC_SCHEDULER_PROFILING=1 | ||
| DYNAMIC_SCHEDULER_USE_INTERNAL_SCHEDULER=0 | ||
| DYNAMIC_SCHEDULER_STOP_SERVICE_TIMEOUT=01:00:00 | ||
| DYNAMIC_SCHEDULER_TRACING=null | ||
| DYNAMIC_SCHEDULER_TRACING={} | ||
| DYNAMIC_SCHEDULER_UI_STORAGE_SECRET=adminadmin | ||
|
|
||
| FUNCTION_SERVICES_AUTHORS='{"UN": {"name": "Unknown", "email": "[email protected]", "affiliation": "unknown"}}' | ||
|
|
@@ -158,13 +158,13 @@ INVITATIONS_PORT=8000 | |
| INVITATIONS_SECRET_KEY='REPLACE_ME_with_result__Fernet_generate_key=' | ||
| INVITATIONS_SWAGGER_API_DOC_ENABLED=1 | ||
| INVITATIONS_USERNAME=admin | ||
| INVITATIONS_TRACING=null | ||
| INVITATIONS_TRACING={} | ||
|
|
||
| LOG_FORMAT_LOCAL_DEV_ENABLED=1 | ||
| LOG_FILTER_MAPPING='{"gunicorn.access":[" /v0/ ", " /v0/health "], "uvicorn.access":[" / "]}' | ||
|
|
||
| NOTIFICATIONS_LOGLEVEL=INFO | ||
| NOTIFICATIONS_TRACING=null | ||
| NOTIFICATIONS_TRACING={} | ||
|
|
||
| PAYMENTS_ACCESS_TOKEN_EXPIRE_MINUTES=30 | ||
| PAYMENTS_ACCESS_TOKEN_SECRET_KEY=2c0411810565e063309be1457009fb39ce023946f6a354e6935107b57676 | ||
|
|
@@ -186,7 +186,7 @@ PAYMENTS_STRIPE_API_SECRET='REPLACE_ME_with_api_secret' | |
| PAYMENTS_STRIPE_URL=https://api.stripe.com | ||
| PAYMENTS_SWAGGER_API_DOC_ENABLED=1 | ||
| PAYMENTS_USERNAME=admin | ||
| PAYMENTS_TRACING=null | ||
| PAYMENTS_TRACING={} | ||
|
|
||
| POSTGRES_DB=simcoredb | ||
| POSTGRES_ENDPOINT=postgres:5432 | ||
|
|
@@ -227,7 +227,7 @@ RESOURCE_USAGE_TRACKER_MISSED_HEARTBEAT_CHECK_ENABLED=1 | |
| RESOURCE_USAGE_TRACKER_MISSED_HEARTBEAT_COUNTER_FAIL=6 | ||
| RESOURCE_USAGE_TRACKER_MISSED_HEARTBEAT_INTERVAL_SEC=300 | ||
| RESOURCE_USAGE_TRACKER_S3=null | ||
| RESOURCE_USAGE_TRACKER_TRACING=null | ||
| RESOURCE_USAGE_TRACKER_TRACING={} | ||
|
|
||
| # NOTE: 172.17.0.1 is the docker0 interface, which redirect from inside a container onto the host network interface. | ||
| R_CLONE_OPTION_BUFFER_SIZE=16M | ||
|
|
@@ -259,7 +259,7 @@ STORAGE_HOST=storage | |
| STORAGE_LOGLEVEL=INFO | ||
| STORAGE_PORT=8080 | ||
| STORAGE_PROFILING=1 | ||
| STORAGE_TRACING=null | ||
| STORAGE_TRACING={} | ||
| # STORAGE ---- | ||
|
|
||
| SWARM_STACK_NAME=master-simcore | ||
|
|
@@ -300,7 +300,7 @@ WB_GC_SOCKETIO=1 | |
| WB_GC_STATICWEB=null | ||
| WB_GC_STUDIES_DISPATCHER=null | ||
| WB_GC_TAGS=0 | ||
| WB_GC_TRACING=null | ||
| WB_GC_TRACING={} | ||
| WB_GC_USERS={} | ||
| WB_GC_WALLETS=0 | ||
|
|
||
|
|
@@ -330,7 +330,7 @@ WB_DB_EL_STATICWEB=null | |
| WB_DB_EL_STORAGE=null | ||
| WB_DB_EL_STUDIES_DISPATCHER=null | ||
| WB_DB_EL_TAGS=0 | ||
| WB_DB_EL_TRACING=null | ||
| WB_DB_EL_TRACING={} | ||
| WB_DB_EL_USERS={} | ||
| WB_DB_EL_WALLETS=0 | ||
|
|
||
|
|
@@ -401,5 +401,5 @@ WEBSERVER_SOCKETIO=1 | |
| WEBSERVER_STATICWEB={} | ||
| WEBSERVER_STUDIES_DISPATCHER={} | ||
| WEBSERVER_TAGS=1 | ||
| WEBSERVER_TRACING=null | ||
| WEBSERVER_TRACING={} | ||
| WEBSERVER_USERS={} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| import pytest | ||
| from opentelemetry.sdk.trace.export import SimpleSpanProcessor | ||
| from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter | ||
| from pytest_mock import MockerFixture | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| async def setup_tracing_fastapi( | ||
| mocker: MockerFixture, monkeypatch: pytest.MonkeyPatch | ||
| ) -> InMemorySpanExporter: | ||
| memory_exporter = InMemorySpanExporter() | ||
| span_processor = SimpleSpanProcessor(memory_exporter) | ||
| mocker.patch( | ||
| "servicelib.fastapi.tracing._create_span_processor", return_value=span_processor | ||
| ) | ||
|
|
||
| monkeypatch.setenv( | ||
| "TRACING_OPENTELEMETRY_COLLECTOR_ENDPOINT", "http://opentelemetry-collector" | ||
| ) | ||
| monkeypatch.setenv("TRACING_OPENTELEMETRY_COLLECTOR_PORT", "4318") | ||
| return memory_exporter | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| async def setup_tracing_aiohttp( | ||
| mocker: MockerFixture, monkeypatch: pytest.MonkeyPatch | ||
| ) -> InMemorySpanExporter: | ||
| memory_exporter = InMemorySpanExporter() | ||
| span_processor = SimpleSpanProcessor(memory_exporter) | ||
| mocker.patch( | ||
| "servicelib.aiohttp.tracing._create_span_processor", return_value=span_processor | ||
| ) | ||
|
|
||
| monkeypatch.setenv( | ||
| "TRACING_OPENTELEMETRY_COLLECTOR_ENDPOINT", "http://opentelemetry-collector" | ||
| ) | ||
| monkeypatch.setenv("TRACING_OPENTELEMETRY_COLLECTOR_PORT", "4318") | ||
| return memory_exporter |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.