Merge pull request #114 from ispras/113-documentation-for-python-package

Author: thientc

.github/workflows/python-tests.yml

@@ -0,0 +1,30 @@
+name: Python Tests
+
+on:
+  push:
+    paths:
+      - 'src/python/**'
+      - '.github/workflows/python-tests.yml'
+  pull_request:
+    paths:
+      - 'src/python/**'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.8', '3.10', '3.12']
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install package with test dependencies
+        run: |
+          cd src/python/futag-package
+          pip install -e ".[test]"
+      - name: Run tests
+        run: |
+          cd src/python/futag-package
+          python -m pytest tests/ -v --tb=short

.github/workflows/syntax-check.yml

@@ -0,0 +1,27 @@
+name: Syntax Check
+
+on: [push, pull_request]
+
+jobs:
+  python-syntax:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Check Python syntax
+        run: |
+          python -c "
+          import ast, sys, pathlib
+          errors = []
+          for f in pathlib.Path('src/python/futag-package/src/futag').glob('*.py'):
+              try:
+                  ast.parse(f.read_text())
+              except SyntaxError as e:
+                  errors.append(f'{f}: {e}')
+          if errors:
+              for e in errors: print(e)
+              sys.exit(1)
+          print('All Python files: syntax OK')
+          "

CHANGELOG.md

@@ -1,48 +1,90 @@
 
 # Change Log
 All notable changes to this project will be documented in this file.
- 
+
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## [3.0.1] - 2025
+
+### Major Refactoring
+- Extracted `BaseGenerator` ABC from 5 duplicated generator classes (12,924 → 4,767 lines)
+- Created `GeneratorState` dataclass replacing 13 mutable instance variables
+- Extracted `BaseFuzzer` from duplicated Fuzzer/NatchFuzzer (1,602 → 891 lines)
+- Moved `ContextGenerator` and `NatchGenerator` to separate modules
+- `FuzzDataProviderGenerator` reduced from 2,715 to 222 lines
+- `BlobStamperGenerator` reduced from 2,693 to 37 lines
+
+### New Features
+- Custom exception hierarchy (`futag.exceptions`: FutagError, InvalidPathError, etc.)
+- Python `logging` module integration (replaces print statements)
+- `GeneratorState.save()`/`restore_from()` for clean recursive backtracking
+- GitHub Actions CI (python-tests.yml, syntax-check.yml)
+
+### Bug Fixes
+- Fixed null-pointer dereference in FutagConsumerAnalyzer (cfg->size() before null check)
+- Fixed memory leak in FutagConsumerAnalyzer (new int() instead of new int[])
+- Fixed `param_list` duplication bug in `__save_old_values`
+- Fixed 14+ file handle leaks (bare open() → with statements)
+- Fixed `_build_ovearall_coverage` typo → `_build_overall_coverage`
+- Replaced bubble sort in `sort_callexprs` with `sorted()`
+
+### Documentation
+- Created docs/architecture.md, docs/generators.md, docs/checkers.md, docs/python-api.md
+- Created CONTRIBUTING.md
+- Added comprehensive docstrings and return type hints to all methods
+- Added GPL v3 license headers to all Python source files
+- Translated template-script.py comments from Russian to English
+
+### C++ Checker Improvements
+- Added `MAX_CFG_BLOCKS` and `REPORT_FILENAME_RAND_LEN` constants
+- Changed `SmallString<0>` to `SmallString<256>`
+- Added Doxygen comments to all checker methods
+- Synchronized base files with LLVM 18 variants
+
 ## 20250824
 - Add support for Fuzzed Data Provider
 
-## 20220716
-- Add modules preprocessor to Futag python-package
-- Fix README of Futag python-package
+## 20230807
+- Optimize ConsumerBuilder
+- Add example for context-generation https://github.com/thientc/Futag-tests/tree/main/json-c-contexts
 
-## 20220727
-- Add custom-llvm: download and build llvm, clang, compiler-rt
-- Fix document
+## 20230711
+- Support generation fuzz driver for Natch data: https://github.com/thientc/Futag-tests/tree/main/Natch
 
-## 20220801
-- Add multi-processing support for compiling
-- TODO: Check analysis result befor generating fuzz-driver
+## 20230522
+- Fix error in generator
+- Add generation for pugi::char_t *&
 
-## 20220808
-- Fix bug in generator
-- Fix for svace analysing
-- add first version of fuzzer and result of Fuzzing for Svace
+## 20230417
+- Add generation for anonymous function
+- Fix error in Builder
 
-## 20220811
-- Fix bug in generator
-- Add pre release package
-- Fix document
+## 20230320
+- Support for context generation
 
-## 20220821
-- Fix bug in generator
-- Add release package
-- Fix document
+## 20230305
+- Fix error python in Builder
+- Fix error python in Generator for wchar_t string
 
-## 20220911
-- Add support for fuzz-introspector
-- Migrate to llvm-14.0.6
+## 20230214
+- Add is_simple for 4consummer_analysis_db
+- Add CFG and DFC analysis
+- Add Fuzzer extra params support
 
-## 20220921
-- Add support for Makefile
-- Generation for global function of C++ libraries
-- Add testing repository: https://github.com/thientc/Futag-tests
+## 20221220
+- Fix errors while compiling AFL++, return coverage parameters
+- Fix Readme
+- change LLVM_ENABLE_ZLIB to ON
+
+## 20221107
+- And generation for anonymous functions
+- Reformat Python classes
+- Fix included paths of compiling command
+
+## 20221018
+- Add support for C++, generate for constructors and for method of class, which has default constructors
+- Tested on FreeImage and Pugixml
 
 ## 20221012
 - Add support for AFLplusplus
@@ -51,43 +93,38 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - Add compilition database of building
 - Add analysis of headers
 
-## 20221018
-- Add support for C++, generate for constructors and for method of class, which has default constructors
-- Tested on FreeImage and Pugixml
-
-## 20221107
-- And generation for anonymous functions
-- Reformat Python classes 
-- Fix included paths of compiling command
-
-## 20221220
-- Fix errors while compiling AFL++, return coverage parameters
-- Fix Readme
-- change LLVM_ENABLE_ZLIB to ON
+## 20220921
+- Add support for Makefile
+- Generation for global function of C++ libraries
+- Add testing repository: https://github.com/thientc/Futag-tests
 
-## 20230214
-- Add is_simple for 4consummer_analysis_db
-- Add CFG and DFC analysis
-- Add Fuzzer extra params support
+## 20220911
+- Add support for fuzz-introspector
+- Migrate to llvm-14.0.6
 
-## 20230320
-- Support for context generation
+## 20220821
+- Fix bug in generator
+- Add release package
+- Fix document
 
-## 20230305
-- Fix error python in Builder
-- Fix error python in Generator for wchar_t string
+## 20220811
+- Fix bug in generator
+- Add pre release package
+- Fix document
 
-## 20230417
-- Add generation for anonymous function
-- Fix error in Builder
+## 20220808
+- Fix bug in generator
+- Fix for svace analysing
+- add first version of fuzzer and result of Fuzzing for Svace
 
-## 20230522
-- Fix error in generator
-- Add generation for pugi::char_t *&
+## 20220801
+- Add multi-processing support for compiling
+- TODO: Check analysis result befor generating fuzz-driver
 
-## 20230711
-- Support generation fuzz driver for Natch data: https://github.com/thientc/Futag-tests/tree/main/Natch
+## 20220727
+- Add custom-llvm: download and build llvm, clang, compiler-rt
+- Fix document
 
-## 20230807
-- Optimize ConsumerBuilder
-- Add example for context-generation https://github.com/thientc/Futag-tests/tree/main/json-c-contexts
+## 20220716
+- Add modules preprocessor to Futag python-package
+- Fix README of Futag python-package

CONTRIBUTING.md

@@ -0,0 +1,66 @@
+# Contributing to Futag
+
+## Development Setup
+
+### Prerequisites
+- CMake >= 3.13.4
+- GCC >= 7.1.0
+- Python >= 3.8
+- pip >= 22.1.1
+
+### Building the LLVM toolchain
+
+```bash
+cd custom-llvm && ./prepare.sh
+cd ../build && ./build.sh
+```
+
+### Installing the Python package (development mode)
+
+```bash
+cd src/python/futag-package
+pip install -e ".[test]"
+```
+
+### Running tests
+
+```bash
+cd src/python/futag-package
+python -m pytest tests/ -v
+```
+
+## Code Style
+
+### Python
+- Follow PEP 8
+- Use Google-style docstrings
+- All new methods must have type hints and docstrings
+- Use `logging` module instead of `print()`
+- Use `with` statements for file I/O
+- Raise exceptions from `futag.exceptions` instead of `sys.exit()`
+
+### C++
+- Follow LLVM coding style (configured in .clang-format)
+- Column limit: 80 characters
+- Indent: 4 spaces
+
+## Pull Request Process
+
+1. Create a feature branch from `main`
+2. Make your changes with clear commit messages
+3. Ensure all Python tests pass
+4. Update documentation if APIs change
+5. Submit PR with description of changes
+
+## Adding a New Generator Backend
+
+See docs/generators.md for the BaseGenerator pattern. Implement 10 abstract `_gen_*` methods.
+
+## LLVM Version Support
+
+When adding support for a new LLVM version:
+1. Create version-specific source files (e.g., FutagAnalyzer19.cpp)
+2. Create version-specific CMakeLists (e.g., CMakeLists19.txt)
+3. Update Checkers.td with version-specific copy
+4. Update build/build.sh version detection
+5. The base file should always match the latest supported LLVM version

How-to-work-with-Futag.md

@@ -164,4 +164,59 @@ json_file="/path/to/analysis/folder/futag-analysis-result.json" #Путь к ф
 g.gen_targets() # генерация фаззинг-оберток
 g.compile_targets( # компиляция фаззинг-оберток
     True, # генерация Makefile
-    16) # количество потоков при компиляции
+    16) # количество потоков при компиляции
+
+## Генераторы фаззинг-оберток
+
+Futag поддерживает несколько вариантов генерации фаззинг-оберток. Все генераторы наследуют от базового класса `BaseGenerator`.
+
+### Generator (стандартный)
+Использует `memcpy()` для чтения данных из буфера. Поддерживает C и C++.
+
+```python
+from futag.generator import Generator
+generator = Generator(futag_llvm_path, library_root)
+generator.gen_targets(max_wrappers=10)
+generator.compile_targets(workers=4)
+```
+
+### FuzzDataProviderGenerator
+Использует API `FuzzedDataProvider` из libFuzzer для типобезопасного чтения данных. Только C++.
+
+```python
+from futag.fdp_generator import FuzzDataProviderGenerator
+generator = FuzzDataProviderGenerator(futag_llvm_path, library_root)
+generator.gen_targets(max_wrappers=100)
+generator.compile_targets(workers=4, keep_failed=True)
+```
+
+### ContextGenerator
+Генерирует обёртки на основе контекстов использования библиотеки в потребительских программах.
+
+```python
+from futag.generator import ContextGenerator
+ctx_gen = ContextGenerator(futag_llvm_path, library_root)
+ctx_gen.gen_context()
+ctx_gen.compile_targets(keep_failed=True)
+```
+
+## Управление логированием
+
+Futag использует модуль `logging` для вывода информации. Настройка уровня логирования:
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)   # Стандартный вывод
+logging.basicConfig(level=logging.DEBUG)  # Подробный вывод
+logging.basicConfig(level=logging.WARNING)  # Тихий режим
+```
+
+## Обработка ошибок
+
+Модуль `futag.exceptions` предоставляет иерархию исключений:
+- `FutagError` — базовое исключение
+- `InvalidPathError` — неверный путь к файлу или директории
+- `InvalidConfigError` — неверная конфигурация
+- `BuildError` — ошибка сборки библиотеки
+- `GenerationError` — ошибка генерации фаззинг-оберток
+- `AnalysisError` — ошибка анализа результатов