Refactor agent startup logic and experimental os.fork support

.github/workflows/CI.yaml

@@ -155,7 +155,7 @@ jobs:
       ( always() && ! cancelled() ) &&
       ((github.event_name == 'schedule' && github.repository == 'apache/skywalking-python') || needs.changes.outputs.agent == 'true')
     runs-on: ubuntu-latest
-    timeout-minutes: 30
+    timeout-minutes: 20
     strategy:
       matrix:
         python-version: [ "3.7", "3.8", "3.9", "3.10", "3.11" ]  

.lift/config.toml

@@ -16,4 +16,4 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-ignoreRules = [ "Unused ignore", "Invalid decoration", "blacklist", "Missing argument", "hardcoded_bind_all_interfaces" ]
+ignoreRules = [ "Unused ignore", "Invalid decoration", "blacklist", "Missing argument", "hardcoded_bind_all_interfaces", "B104", "B201" ]

CHANGELOG.md

@@ -3,12 +3,11 @@
 ### 1.0.0
 
 - **Important Note and Breaking Changes:**
-  - Python 3.6 is no longer supported and may not function properly, Python 3.11 support is added and tested.
-  - A number of common configuration options (environment variables) are renamed to follow the convention of Java agent,
+  - **BREAKING**: Python 3.6 is no longer supported and may not function properly, Python 3.11 support is added and tested.
+  - **BREAKING**: A number of common configuration options and environment variables are renamed to follow the convention of Java agent,
   please check with the latest official documentation before upgrading. (#273, #282)
-
   https://skywalking.apache.org/docs/skywalking-python/v1.0.0/en/setup/configuration/
-  - All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
+  - **BREAKING**: All agent core capabilities are now covered by test cases and enabled by default (Trace, Log, PVM runtime metrics, Profiler)
 
 
 - Feature:
@@ -20,6 +19,7 @@
   - Add support for the tags of Virtual Cache for Redis (#263)
   - Add a new configuration `kafka_namespace` to prefix the kafka topic names (#277)
   - Add log reporter support for loguru (#276)
+  - Add **experimental** support for explicit os.fork(), restarts agent in new process (#286)
 
 - Plugins:
   - Add aioredis, aiormq, amqp, asyncpg, aio-pika, kombu RMQ plugins (#230 Missing test coverage) 

README.md

@@ -2,7 +2,7 @@
 
 <img src="http://skywalking.apache.org/assets/logo.svg" alt="Sky Walking logo" height="90px" align="right" />
 
-**SkyWalking-Python**: The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging abilities for Python projects.
+**SkyWalking-Python**: The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging/profiling abilities for Python projects.
 
 **[SkyWalking](https://github.com/apache/skywalking)**: Application performance monitor tool for distributed systems, especially designed for microservices, cloud native and container-based (Kubernetes) architectures.
 

demo/README.md

@@ -0,0 +1,8 @@
+# Manual Test
+
+Edge cases on advanced features would benefit from a manual testing process.
+
+This directory holds some utils and scripts that are convenient for such use cases.
+
+## Docker-compose.yaml
+This docker-compose.yaml spins up a fresh Apache SkyWalking instance along with UI (localhost:8080) and SW_CTL CLI for you to verify.

demo/__init__.py

@@ -0,0 +1,14 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

demo/docker-compose.yaml

@@ -0,0 +1,106 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+services:
+  oap:
+    container_name: oap
+    image: apache/skywalking-oap-server:9.3.0
+    # Python agent supports gRPC/ HTTP/ Kafka reporting
+    expose:
+      - 11800 # gRPC
+      - 12800 # HTTP
+    networks:
+      - manual
+    environment:
+      SW_KAFKA_FETCHER: default
+      SW_KAFKA_FETCHER_SERVERS: kafka:9092
+      SW_KAFKA_FETCHER_PARTITIONS: 2
+      SW_KAFKA_FETCHER_PARTITIONS_FACTOR: 1
+    healthcheck:
+      test: [ "CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/11800" ]
+      interval: 5s
+      timeout: 60s
+      retries: 120
+    ports:
+      - "12800:12800"
+      - "11800:11800"
+    depends_on:
+      - kafka
+
+
+  ui:
+    image: apache/skywalking-ui:9.3.0
+    container_name: ui
+    depends_on:
+      oap:
+        condition: service_healthy
+    networks:
+      - manual
+    ports:
+      - "8080:8080"
+    environment:
+      SW_OAP_ADDRESS: "http://oap:12800"
+
+  zookeeper:
+    container_name: zk
+    image: confluentinc/cp-zookeeper:latest
+    ports:
+      - "2181:2181"
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_TICK_TIME: 2000
+    networks:
+      - manual
+
+  kafka:
+    container_name: kafka
+    image: confluentinc/cp-kafka
+    expose:
+      - 9092
+      - 9094
+    ports:
+      - 9092:9092
+      - 9094:9094
+    depends_on:
+      - zookeeper
+    environment:
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+      KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+    networks:
+      - manual
+
+#
+#  kafka-ui:
+#    image: provectuslabs/kafka-ui
+#    container_name: kafka-ui
+#    ports:
+#      - "8088:8080"
+#    restart: always
+#    environment:
+#      - KAFKA_CLUSTERS_0_NAME=local
+#      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
+#    depends_on:
+#      - kafka
+#    networks:
+#      - manual
+
+networks:
+  manual:

demo/flask_consumer_fork.py

@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+from flask import Flask
+from skywalking import agent, config
+import requests
+
+# Profiling only available in gRPC, meter only in kafka + grpc
+config.init(agent_collector_backend_services='localhost:11800', agent_protocol='grpc',
+            agent_name='great-app-consumer-grpc',
+            kafka_bootstrap_servers='localhost:9094',  # If you use kafka, set this
+            agent_instance_name='instance-01',
+            agent_experimental_fork_support=True, agent_logging_level='DEBUG', agent_log_reporter_active=True,
+            agent_meter_reporter_active=True,
+            agent_profile_active=True)
+
+agent.start()
+
+parent_pid = os.getpid()
+pid = os.fork()
+
+app = Flask(__name__)
+
+
+@app.route('/', methods=['POST', 'GET'])
+def application():
+    res = requests.get('http://localhost:9999')
+    return res.json()
+
+
+if __name__ == '__main__':
+    PORT = 9097 if pid == 0 else 9098  # 0 is child process
+    app.run(host='0.0.0.0', port=PORT, debug=False)  # RELOADER IS ALSO FORKED

demo/flask_provider_single.py

@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from flask import Flask, jsonify
+from skywalking import agent, config
+
+config.init(agent_collector_backend_services='localhost:11800', agent_protocol='grpc',
+            agent_name='great-app-provider-grpc',
+            kafka_bootstrap_servers='localhost:9094',  # If you use kafka, set this
+            agent_instance_name='instance-01',
+            agent_experimental_fork_support=True,
+            agent_logging_level='DEBUG',
+            agent_log_reporter_active=True,
+            agent_meter_reporter_active=True,
+            agent_profile_active=True)
+
+
+agent.start()
+
+app = Flask(__name__)
+
+
+@app.route('/', methods=['POST', 'GET'])
+def application():
+    return jsonify({'status': 'ok'})
+
+
+if __name__ == '__main__':
+    app.run(host='0.0.0.0', port=9999, debug=True, use_reloader=False)

docs/README.md

@@ -2,7 +2,7 @@
 
 **This is the official documentation of SkyWalking Python agent. Welcome to the SkyWalking community!**
 
-The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging abilities for Python projects.
+The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging/profiling abilities for Python projects.
 
 This documentation covers a number of ways to set up the Python agent for various use cases.
 

docs/en/contribution/How-to-release.md

@@ -201,7 +201,7 @@ Vote result should follow these:
 
     On behalf of the SkyWalking Team, I’m glad to announce that SkyWalking Python $VERSION is now released.
 
-    SkyWalking Python: The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging abilities for Python projects.
+    SkyWalking Python: The Python Agent for Apache SkyWalking provides the native tracing/metrics/logging/profiling abilities for Python projects.
 
     SkyWalking: APM (application performance monitor) tool for distributed systems, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.
 

docs/en/setup/CLI.md

@@ -1,9 +1,12 @@
-# SkyWalking Python Agent Command-Line Interface(CLI)
+# SkyWalking Python Agent Command-Line Interface (sw-python CLI)
 
 In releases before 0.7.0, you would at least need to add the following lines to your applications to get the agent attached and running.  
 
+This is the recommended way of running your application with Python agent.
+
 ```python
-from skywalking import agent
+from skywalking import agent, config
+config.init(SomeConfig)
 agent.start()
 ```
 
@@ -17,6 +20,8 @@ just like the [SkyWalking Java Agent](https://github.com/apache/skywalking-java)
 Upon successful [installation of the SkyWalking Python agent via pip](Installation.md#from-pypi),
 a command-line script `sw-python` is installed in your environment (virtual env preferred).
 
+run `sw-python` to see if it is available.
+
 ### The `run` option
 
 Currently, the `sw-python` CLI provides a `run` option, which you can use to execute your applications
@@ -64,7 +69,7 @@ You would normally want to provide additional configurations other than the defa
 The currently supported method is to provide the environment variables listed 
 and explained in the [Environment Variables List](Configuration.md).
 
-#### Through a sw-config.yaml
+#### Through a sw-config.yaml (TBD)
 
 Currently, only environment variable configuration is supported; an optional `yaml` configuration is to be implemented.
 

docs/en/setup/Configuration.md

@@ -18,7 +18,7 @@ export SW_AGENT_YourConfiguration=YourValue
 | Configuration | Environment Variable | Type | Default Value | Description |
 | :------------ | :------------ | :------------ | :------------ | :------------ |
 | agent_collector_backend_services | SW_AGENT_AGENT_COLLECTOR_BACKEND_SERVICES | <class 'str'> | oap_host:oap_port | The backend OAP server address, 11800 is default OAP gRPC port, 12800 is HTTP, Kafka ignores this option and uses kafka_bootstrap_servers option. **This option should be changed accordingly with selected protocol** |
-| protocol | SW_AGENT_PROTOCOL | <class 'str'> | grpc | The protocol to communicate with the backend OAP, `http`, `grpc` or `kafka`, **we highly suggest using `grpc` in production as it's well optimized than `http`**. The `kafka` protocol provides an alternative way to submit data to the backend. |
+| agent_protocol | SW_AGENT_AGENT_PROTOCOL | <class 'str'> | grpc | The protocol to communicate with the backend OAP, `http`, `grpc` or `kafka`, **we highly suggest using `grpc` in production as it's well optimized than `http`**. The `kafka` protocol provides an alternative way to submit data to the backend. |
 | agent_name | SW_AGENT_AGENT_NAME | <class 'str'> | Python Service Name | The name of your awesome Python service |
 | agent_instance_name | SW_AGENT_AGENT_INSTANCE_NAME | <class 'str'> | str(uuid.uuid1()).replace('-', '') | The name of this particular awesome Python service instance |
 | agent_namespace | SW_AGENT_AGENT_NAMESPACE | <class 'str'> |  | The agent namespace of the Python service (available as tag and the suffix of service name) |

docs/en/setup/Installation.md

@@ -6,7 +6,7 @@ You can install the SkyWalking Python agent via various ways described next.
 
 > **Already installed? Check out easy ways to start the agent in your application**
 
-> [Non-intrusive](CLI.md) | [Intrusive <minimal>](Intrusive.md) | [Containerization](Container.md) 
+> [Non-intrusive <Recommended>](CLI.md) | [Intrusive <minimal>](Intrusive.md) | [Containerization](Container.md) 
 
 > **All available configurations are listed [here](Configuration.md)**
 
@@ -36,6 +36,9 @@ from where you can use `pip` to install:
 # Install the latest version, using the default gRPC protocol to report data to OAP
 pip install "apache-skywalking"
 
+# Install support for every protocol (gRPC, HTTP, Kafka)
+pip install "apache-skywalking[all]"
+
 # Install the latest version, using the http protocol to report data to OAP
 pip install "apache-skywalking[http]"