Skip to content

Commit d5a9506

Browse files
SuperskyyySuperskyyy
authored
Refactor agent core startup solo (#89) (#287)
This PR is made of two tightly coupled parts: * Total rewrite of agent startup logic from module functions -> singleton class. (some other logic was changed in meter to fix wrong forking behavior) * Provide experimental support for os.fork(), exposed as an option. * A demo directory to provide easier access to oap/kafka/demoservices (for contributors). Minor changes: * Docs: fixed some missed ones over time. * Fixed a redis bug.
1 parent 3f2926b commit d5a9506

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+706
-428
lines changed

‎.github/workflows/CI.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ jobs:
155155
( always() && ! cancelled() ) &&
156156
((github.event_name == 'schedule' && github.repository == 'apache/skywalking-python') || needs.changes.outputs.agent == 'true')
157157
runs-on: ubuntu-latest
158-
timeout-minutes: 30
158+
timeout-minutes: 20
159159
strategy:
160160
matrix:
161161
python-version: [ "3.7", "3.8", "3.9", "3.10", "3.11" ]

‎.lift/config.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,4 @@
1616
# specific language governing permissions and limitations
1717
# under the License.
1818
#
19-
ignoreRules = [ "Unused ignore", "Invalid decoration", "blacklist", "Missing argument", "hardcoded_bind_all_interfaces" ]
19+
ignoreRules = [ "Unused ignore", "Invalid decoration", "blacklist", "Missing argument", "hardcoded_bind_all_interfaces", "B104", "B201" ]

‎CHANGELOG.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,11 @@
33
### 1.0.0
44

55
- **Important Note and Breaking Changes:**
6-
- Python 3.6 is no longer supported and may not function properly, Python 3.11 support is added and tested.
7-
- A number of common configuration options (environment variables) are renamed to follow the convention of Java agent,
6+
- **BREAKING**: Python 3.6 is no longer supported and may not function properly, Python 3.11 support is added and tested.
7+
- **BREAKING**: A number of common configuration options and environment variables are renamed to follow the convention of Java agent,
88
please check with the latest official documentation before upgrading. (#273, #282)
9-
109
https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/
11-
- All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
10+
- **BREAKING**: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
1211

1312

1413
- Feature:
@@ -20,6 +19,7 @@
2019
- Add support for the tags of Virtual Cache for Redis (#263)
2120
- Add a new configuration `kafka_namespace` to prefix the kafka topic names (#277)
2221
- Add log reporter support for loguru (#276)
22+
- Add **experimental** support for explicit os.fork(), restarts agent in new process (#286)
2323

2424
- Plugins:
2525
- Add aioredis, aiormq, amqp, asyncpg, aio-pika, kombu RMQ plugins (#230 Missing test coverage)

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
<img src="http://skywalking.apache.org/assets/logo.svg" alt="Sky Walking logo" height="90px" align="right" />
44

5-
**SkyWalking-Python**: The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging abilities for Python projects.
5+
**SkyWalking-Python**: The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging/profiling abilities for Python projects.
66

77
**[SkyWalking](https://github.com/apache/skywalking)**: Application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures.
88

‎demo/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Manual Test
2+
3+
Edge cases on advanced features would benefit from a manual testing process.
4+
5+
This directory holds some utils and scripts that are convenient for such use cases.
6+
7+
## Docker-compose.yaml
8+
This docker-compose.yaml spins up a fresh Apache SkyWalking instance along with UI (localhost:8080) and SW_CTL CLI for you to verify.

‎demo/__init__.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.

‎demo/docker-compose.yaml‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
services:
19+
oap:
20+
container_name: oap
21+
image: apache/skywalking-oap-server:9.3.0
22+
# Python agent supports gRPC/ HTTP/ Kafka reporting
23+
expose:
24+
- 11800 # gRPC
25+
- 12800 # HTTP
26+
networks:
27+
- manual
28+
environment:
29+
SW_KAFKA_FETCHER: default
30+
SW_KAFKA_FETCHER_SERVERS: kafka:9092
31+
SW_KAFKA_FETCHER_PARTITIONS: 2
32+
SW_KAFKA_FETCHER_PARTITIONS_FACTOR: 1
33+
healthcheck:
34+
test: [ "CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/11800" ]
35+
interval: 5s
36+
timeout: 60s
37+
retries: 120
38+
ports:
39+
- "12800:12800"
40+
- "11800:11800"
41+
depends_on:
42+
- kafka
43+
44+
45+
ui:
46+
image: apache/skywalking-ui:9.3.0
47+
container_name: ui
48+
depends_on:
49+
oap:
50+
condition: service_healthy
51+
networks:
52+
- manual
53+
ports:
54+
- "8080:8080"
55+
environment:
56+
SW_OAP_ADDRESS: "http://oap:12800"
57+
58+
zookeeper:
59+
container_name: zk
60+
image: confluentinc/cp-zookeeper:latest
61+
ports:
62+
- "2181:2181"
63+
environment:
64+
ZOOKEEPER_CLIENT_PORT: 2181
65+
ZOOKEEPER_TICK_TIME: 2000
66+
networks:
67+
- manual
68+
69+
kafka:
70+
container_name: kafka
71+
image: confluentinc/cp-kafka
72+
expose:
73+
- 9092
74+
- 9094
75+
ports:
76+
- 9092:9092
77+
- 9094:9094
78+
depends_on:
79+
- zookeeper
80+
environment:
81+
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
82+
KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094
83+
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
84+
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
85+
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
86+
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
87+
networks:
88+
- manual
89+
90+
#
91+
# kafka-ui:
92+
# image: provectuslabs/kafka-ui
93+
# container_name: kafka-ui
94+
# ports:
95+
# - "8088:8080"
96+
# restart: always
97+
# environment:
98+
# - KAFKA_CLUSTERS_0_NAME=local
99+
# - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
100+
# depends_on:
101+
# - kafka
102+
# networks:
103+
# - manual
104+
105+
networks:
106+
manual:

‎demo/flask_consumer_fork.py‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
import os
17+
18+
from flask import Flask
19+
from skywalking import agent, config
20+
import requests
21+
22+
# Profiling only available in gRPC, meter only in kafka + grpc
23+
config.init(agent_collector_backend_services='localhost:11800', agent_protocol='grpc',
24+
agent_name='great-app-consumer-grpc',
25+
kafka_bootstrap_servers='localhost:9094', # If you use kafka, set this
26+
agent_instance_name='instance-01',
27+
agent_experimental_fork_support=True, agent_logging_level='DEBUG', agent_log_reporter_active=True,
28+
agent_meter_reporter_active=True,
29+
agent_profile_active=True)
30+
31+
agent.start()
32+
33+
parent_pid = os.getpid()
34+
pid = os.fork()
35+
36+
app = Flask(__name__)
37+
38+
39+
@app.route('/', methods=['POST', 'GET'])
40+
def application():
41+
res = requests.get('http://localhost:9999')
42+
return res.json()
43+
44+
45+
if __name__ == '__main__':
46+
PORT = 9097 if pid == 0 else 9098 # 0 is child process
47+
app.run(host='0.0.0.0', port=PORT, debug=False) # RELOADER IS ALSO FORKED

‎demo/flask_provider_single.py‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one or more
2+
# contributor license agreements. See the NOTICE file distributed with
3+
# this work for additional information regarding copyright ownership.
4+
# The ASF licenses this file to You under the Apache License, Version 2.0
5+
# (the "License"); you may not use this file except in compliance with
6+
# the License. You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
from flask import Flask, jsonify
16+
from skywalking import agent, config
17+
18+
config.init(agent_collector_backend_services='localhost:11800', agent_protocol='grpc',
19+
agent_name='great-app-provider-grpc',
20+
kafka_bootstrap_servers='localhost:9094', # If you use kafka, set this
21+
agent_instance_name='instance-01',
22+
agent_experimental_fork_support=True,
23+
agent_logging_level='DEBUG',
24+
agent_log_reporter_active=True,
25+
agent_meter_reporter_active=True,
26+
agent_profile_active=True)
27+
28+
29+
agent.start()
30+
31+
app = Flask(__name__)
32+
33+
34+
@app.route('/', methods=['POST', 'GET'])
35+
def application():
36+
return jsonify({'status': 'ok'})
37+
38+
39+
if __name__ == '__main__':
40+
app.run(host='0.0.0.0', port=9999, debug=True, use_reloader=False)

‎docs/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
**This is the official documentation of SkyWalking Python agent. Welcome to the SkyWalking community!**
44

5-
The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging abilities for Python projects.
5+
The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging/profiling abilities for Python projects.
66

77
This documentation covers a number of ways to set up the Python agent for various use cases.
88

0 commit comments

Comments
 (0)