Merge pull request #3 from kangtegong/feature/python3.10+

Feature/python3.10+

Author: kangtegong

.github/workflows/push-test.yml

@@ -6,11 +6,8 @@ jobs:
   test:
     strategy:
       matrix:
-        os: [ubuntu-latest, windows-latest, macos-latest]
-        python-version: ["3.12", "3.13"]
-        exclude:
-          - os: windows-latest
-            python-version: "3.13"
+        os: [ubuntu-latest]
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
     runs-on: ${{ matrix.os }}
 
     steps:
@@ -22,11 +19,6 @@ jobs:
       with:
         python-version: ${{ matrix.python-version }}
         check-latest: true
-
-    - name: Install dependencies
-      if: runner.os == 'Windows'
-      run: pip install -r requirements.txt
-   
     - name: Run tests
       run: |
         python tests/runtests.py

.github/workflows/python-publish.yml

@@ -25,7 +25,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v3
       with:
-        python-version: '3.12'
+        python-version: '3.8'
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip

README-ko.md

@@ -2,7 +2,7 @@
 
 ## Introduction
 
-**pyftrace**는 Python 스크립트 내에서 함수 호출을 모니터링할 수 있는 경량 Python 함수 추적 도구입니다. Python 3.12의 `sys.monitoring`을 활용하여 Python 이벤트를 모니터링하고, 모니터링 결과를 기반으로 함수 호출/리턴을 추적합니다. pyftrace를 사용하면 여러 모듈에 걸친 함수 호출을 추적할 수 있고, 호출 계층 구조를 시각적으로 나타낼 수 있으며, 추적 결과 보고서를 생성할 수 있습니다.
+**pyftrace**는 Python 스크립트 내에서 함수 호출을 모니터링할 수 있는 경량 Python 함수 추적 도구입니다. Python 3.8 ~3.11 버전에서는 `sys.setprofile`을,  Python 3.12 이상 버전에서는 `sys.monitoring`을 활용하여 Python 이벤트를 모니터링하고, 모니터링 결과를 기반으로 함수 호출/리턴을 추적합니다. pyftrace를 사용하면 여러 모듈에 걸친 함수 호출을 추적할 수 있고, 호출 계층 구조를 시각적으로 나타낼 수 있으며, 추적 결과 보고서를 생성할 수 있습니다.
 
 ![pyftrace-demo](assets/pyftrace-demo.gif)
 
@@ -36,7 +36,7 @@ options:
 
 ### 요구 사항
 
-- **Python 버전**: pyftrace는 Python 3.12 이상이 필요합니다. Python 3.12에 도입된 새로운 `sys.monitoring` 모듈을 사용하기 때문입니다.
+- **Python 버전**: pyftrace는 Python 3.8 이상이 필요합니다.
 
 ```bash
 $ pyftrace [options] /path/to/python/script.py

README.md

@@ -7,7 +7,7 @@
 
 ## Introduction
 
-**pyftrace** is a lightweight Python function tracing tool designed to monitor and report on function calls within Python scripts. It leverages Python 3.12's sys.monitoring to monitor Python events and trace functions based on the results. With pyftrace, you can trace function calls across multiple modules, visualize call hierarchies, and generate execution time reports.
+**pyftrace** is a lightweight Python function tracing tool designed to monitor and report on function calls within Python scripts. It leverages sys.setprofile(Python 3.8 ~ 3.11), sys.monitoring(Python 3.12 ~) to monitor Python events and trace functions based on the results. With pyftrace, you can trace function calls across multiple modules, visualize call hierarchies, and generate execution time reports.
 
 ![pyftrace-demo](assets/pyftrace-demo.gif)
 
@@ -41,7 +41,7 @@ options:
 
 ### Requirements
 
-- **Python Version**: pyftrace requires **Python 3.12** or higher due to its use of the new `sys.monitoring` module introduced in Python 3.12.
+- **Python Version**: pyftrace requires **Python 3.8+**.
 
 ```bash
 $ pyftrace [options] /path/to/python/script.py

TODO

@@ -1,13 +1,9 @@
-- add -vv option for large codebase (e.g. Django, Pytorch, etc)
-  - -v option: Additionally enable built-in functions tracing
-  - -vv option: Additionally enable stdlib tracing 
-    - stdlib path: sysconfig.get_paths()["stdlib"]
+- depth limit: --depth/-D <int> 
 - trace format using ├, │, ─ 
 - tui: fold/unfold
 - tui: '/' (search function)
 - function filtering: --function/-f <string>
 - function-exclude filtering: --exclude-function/-e <string>
-- depth limit: --depth/-D <int> 
 - output format: -o/--output -O <filename> / sqlite dump
 - multiprocessing(pid) & multithreading(tid) tracing
 - argument tracing

examples/requests_example.py

@@ -0,0 +1,6 @@
+import requests
+
+x = requests.get('https://example.com')
+
+print(x.text)
+

examples/torch_example.py

@@ -0,0 +1,59 @@
+# Example from: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html
+# -*- coding: utf-8 -*-
+import torch
+import math
+
+dtype = torch.float
+device = "cuda" if torch.cuda.is_available() else "cpu"
+torch.set_default_device(device)
+
+# Create Tensors to hold input and outputs.
+# By default, requires_grad=False, which indicates that we do not need to
+# compute gradients with respect to these Tensors during the backward pass.
+x = torch.linspace(-math.pi, math.pi, 2000, dtype=dtype)
+y = torch.sin(x)
+
+# Create random Tensors for weights. For a third order polynomial, we need
+# 4 weights: y = a + b x + c x^2 + d x^3
+# Setting requires_grad=True indicates that we want to compute gradients with
+# respect to these Tensors during the backward pass.
+a = torch.randn((), dtype=dtype, requires_grad=True)
+b = torch.randn((), dtype=dtype, requires_grad=True)
+c = torch.randn((), dtype=dtype, requires_grad=True)
+d = torch.randn((), dtype=dtype, requires_grad=True)
+
+learning_rate = 1e-6
+for t in range(100):
+    # Forward pass: compute predicted y using operations on Tensors.
+    y_pred = a + b * x + c * x ** 2 + d * x ** 3
+
+    # Compute and print loss using operations on Tensors.
+    # Now loss is a Tensor of shape (1,)
+    # loss.item() gets the scalar value held in the loss.
+    loss = (y_pred - y).pow(2).sum()
+    if t % 100 == 99:
+        print(t, loss.item())
+
+    # Use autograd to compute the backward pass. This call will compute the
+    # gradient of loss with respect to all Tensors with requires_grad=True.
+    # After this call a.grad, b.grad. c.grad and d.grad will be Tensors holding
+    # the gradient of the loss with respect to a, b, c, d respectively.
+    loss.backward()
+
+    # Manually update weights using gradient descent. Wrap in torch.no_grad()
+    # because weights have requires_grad=True, but we don't need to track this
+    # in autograd.
+    with torch.no_grad():
+        a -= learning_rate * a.grad
+        b -= learning_rate * b.grad
+        c -= learning_rate * c.grad
+        d -= learning_rate * d.grad
+
+        # Manually zero the gradients after updating weights
+        a.grad = None
+        b.grad = None
+        c.grad = None
+        d.grad = None
+
+print(f'Result: y = {a.item()} + {b.item()} x + {c.item()} x^2 + {d.item()} x^3')
+

pyftrace/__init__.py

@@ -1,3 +1,5 @@
 __version__ = "0.2.0"
 
-from .tracer import Pyftrace
+from .tracer import get_tracer
+
+__all__ = ['get_tracer']

pyftrace/engine/__init__.py

@@ -0,0 +1 @@
+

pyftrace/engine/pyftrace_monitoring.py

@@ -0,0 +1,228 @@
+import sys
+import os
+import time
+import weakref
+import sysconfig
+from ..tracer import PyftraceBase
+from ..utils import get_site_packages_modules, resolve_filename, get_line_number
+
+class PyftraceMonitoring(PyftraceBase):
+    """
+    sys.monitoring based tracer.
+    """
+    def setup_tracing(self):
+        self.tool_id = 1
+        sys.monitoring.use_tool_id(self.tool_id, "pyftrace")
+        sys.monitoring.register_callback(self.tool_id, sys.monitoring.events.CALL, self.monitor_call)
+        sys.monitoring.register_callback(self.tool_id, sys.monitoring.events.PY_RETURN, self.monitor_py_return)
+        sys.monitoring.register_callback(self.tool_id, sys.monitoring.events.C_RETURN, self.monitor_c_return)
+        sys.monitoring.register_callback(self.tool_id, sys.monitoring.events.C_RAISE, self.monitor_c_raise)
+        sys.monitoring.set_events(
+            self.tool_id,
+            sys.monitoring.events.CALL |
+            sys.monitoring.events.PY_RETURN |
+            sys.monitoring.events.C_RETURN |
+            sys.monitoring.events.C_RAISE
+        )
+
+    def cleanup_tracing(self):
+        sys.monitoring.free_tool_id(self.tool_id)
+        self.output_stream = None
+
+    def run_python_script(self, script_path, script_args):
+        if self.output_stream:
+            print(f"Running script: {script_path}", file=self.output_stream)
+
+        self.script_name = os.path.abspath(script_path)
+        self.script_dir = os.path.dirname(self.script_name)
+
+        with open(script_path, "r") as file:
+            script_code = file.read()
+            code_object = compile(script_code, script_path, 'exec')
+
+        old_sys_path = sys.path.copy()
+        old_sys_argv = sys.argv.copy()
+        sys.path.insert(0, self.script_dir)
+        sys.argv = [script_path] + script_args
+
+        self.tracing_started = False
+
+        self.setup_tracing()
+
+        try:
+            exec(code_object, {"__file__": script_path, "__name__": "__main__"})
+        finally:
+            self.cleanup_tracing()
+            sys.path = old_sys_path
+            sys.argv = old_sys_argv
+
+    def monitor_call(self, code, instruction_offset, callable_obj, arg0):
+        self.handle_call_event(code, instruction_offset, callable_obj)
+
+    def monitor_py_return(self, code, instruction_offset, retval):
+        self.handle_py_return_event(code, instruction_offset, retval)
+
+    def monitor_c_return(self, code, instruction_offset, callable_obj, arg0):
+        self.handle_c_return_event(code, instruction_offset, callable_obj)
+
+    def monitor_c_raise(self, code, instruction_offset, callable_obj, arg0):
+        pass  # Placeholder for handling C_RAISE events
+
+    def handle_call_event(self, code, instruction_offset, callable_obj):
+        if not self.tracing_started:
+            # Start tracing when  enter script's '<module>' code
+            if code and os.path.abspath(code.co_filename) == os.path.abspath(self.script_name) and code.co_name == '<module>':
+                self.tracing_started = True
+            else:
+                return
+
+        call_lineno = get_line_number(code, instruction_offset)
+        call_filename = resolve_filename(code, None)
+        if call_filename:
+            call_filename = os.path.abspath(call_filename)
+
+        if isinstance(callable_obj, weakref.ReferenceType):
+            callable_obj = callable_obj()
+
+        func_name = getattr(callable_obj, '__name__', str(callable_obj))
+        module_name = getattr(callable_obj, '__module__', None)
+        is_builtin = module_name in (None, 'builtins')
+
+        # Exclude stdlib and frozen modules
+        def_filename = ''
+        func_def_lineno = ''
+        trace_this = False
+
+        if hasattr(callable_obj, '__code__'):
+            func_def_lineno = callable_obj.__code__.co_firstlineno
+            def_filename = os.path.abspath(callable_obj.__code__.co_filename)
+            if not self.is_stdlib_code(def_filename):
+                trace_this = self.should_trace(def_filename) or self.verbose
+            else:
+                trace_this = False  # Exclude stdlib
+        else:
+            def_filename = resolve_filename(None, callable_obj)
+            if def_filename:
+                def_filename = os.path.abspath(def_filename)
+            if is_builtin:
+                # Only trace built-in functions for `verbose`
+                if self.verbose and self.should_trace(call_filename):
+                    trace_this = True
+                else:
+                    trace_this = False
+            else:
+                if self.verbose and self.should_trace(def_filename):
+                    trace_this = True
+
+        if trace_this and not self.is_tracer_code(call_filename):
+            indent = "    " * self.current_depth()
+            if self.show_path:
+                if is_builtin or not def_filename:
+                    func_location = f"{func_name}@{module_name or '<builtin>'}"
+                else:
+                    func_location = f"{func_name}@{def_filename}:{func_def_lineno}"
+                call_location = f"from {call_filename}:{call_lineno}"
+            else:
+                func_location = func_name
+                call_location = f"from line {call_lineno}"
+            if not self.report_mode and self.output_stream:
+                print(f"{indent}Called {func_location} {call_location}", file=self.output_stream)
+            self.call_stack.append((func_name, is_builtin))
+            if self.report_mode:
+                start_time = time.time()
+                if func_name in self.execution_report:
+                    _, total_time, call_count = self.execution_report[func_name]
+                    self.execution_report[func_name] = (start_time, total_time, call_count + 1)
+                else:
+                    self.execution_report[func_name] = (start_time, 0, 1)
+
+    def handle_py_return_event(self, code, instruction_offset, retval):
+        if not self.tracing_started:
+            return
+
+        filename = resolve_filename(code, None)
+        if filename:
+            filename = os.path.abspath(filename)
+        func_name = code.co_name if code else "<unknown>"
+
+        # Skip tracing the '<module>' function's return event
+        if func_name == '<module>':
+            return
+
+        trace_this = self.should_trace(filename) or self.verbose
+
+        if trace_this and not self.is_tracer_code(filename):
+            if self.call_stack:
+                stack_func_name, _ = self.call_stack[-1]
+            else:
+                stack_func_name = "<unknown>"
+
+            indent = "    " * (self.current_depth() - 1)
+
+            if self.show_path:
+                file_info = f" @ {filename}" if filename else ""
+            else:
+                file_info = ""
+
+            if stack_func_name == func_name:
+                if not self.report_mode and self.output_stream:
+                    print(f"{indent}Returning {func_name}-> {retval}{file_info}", file=self.output_stream)
+
+                if self.report_mode and func_name in self.execution_report:
+                    start_time, total_time, call_count = self.execution_report[func_name]
+                    exec_time = time.time() - start_time
+                    self.execution_report[func_name] = (start_time, total_time + exec_time, call_count)
+
+                if self.call_stack and self.call_stack[-1][0] == func_name:
+                    self.call_stack.pop()
+
+    def handle_c_return_event(self, code, instruction_offset, callable_obj):
+        if not self.tracing_started:
+            return
+
+        func_name = getattr(callable_obj, '__name__', str(callable_obj))
+        module_name = getattr(callable_obj, '__module__', None)
+        is_builtin = module_name in (None, 'builtins')
+        filename = resolve_filename(code, callable_obj)
+        if filename:
+            filename = os.path.abspath(filename)
+
+        # Exclude stdlib and frozen modules
+        if self.is_stdlib_code(filename):
+            return
+
+        trace_this = False
+        if is_builtin:
+            # Only trace built-in functions if verbose and called from script
+            if self.verbose and self.call_stack:
+                # Check if the caller is from script
+                caller_filename = filename
+                if caller_filename and self.should_trace(caller_filename):
+                    trace_this = True
+        else:
+            if self.verbose and self.should_trace(filename):
+                trace_this = True
+
+        if trace_this and not self.is_tracer_code(filename):
+            if self.call_stack:
+                stack_func_name, _ = self.call_stack[-1]
+            else:
+                stack_func_name = "<unknown>"
+
+            indent = "    " * (self.current_depth() - 1)
+
+            if self.show_path:
+                file_info = f" @ {filename}" if filename else ""
+            else:
+                file_info = ""
+
+            if stack_func_name == func_name:
+                if not self.report_mode and self.output_stream:
+                    print(f"{indent}Returning {func_name}{file_info}", file=self.output_stream)
+                if self.report_mode and func_name in self.execution_report:
+                    start_time, total_time, call_count = self.execution_report[func_name]
+                    exec_time = time.time() - start_time
+                    self.execution_report[func_name] = (start_time, total_time + exec_time, call_count)
+                if self.call_stack and self.call_stack[-1][0] == func_name:
+                    self.call_stack.pop()
+