Skip to content

Commit 496dacb

Browse files
author
GitHub Copilot
committed
Add scan_bug_history.py top-20 GitHub repo scanning and fix_from_bug_history.py Copilot calibration
- scan_bug_history.py: auto-fetch top N Python repos from GitHub Search API, shallow-clone each, scan 1000 commits per repo, produce combined JSON+MD report - fix_from_bug_history.py: parse bug_history.md, verify each BUG_INTRODUCED/ BUG_FIXED event via GitHub Models API (gh auth token), compute per-bug-type precision, generate and optionally apply patches to a3_python/unsafe/ checkers - results/a3_calibration.md: initial calibration run (41% overall precision; NAME_ERROR 100%, TYPE_CONFUSION 40%, BOUNDS/DIV_ZERO 0%)
1 parent 9a7f5f6 commit 496dacb

348 files changed

Lines changed: 26340 additions & 3084 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/agential-ci.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Agential Demo CI
2+
3+
on:
4+
push:
5+
branches: [ main, master ]
6+
pull_request:
7+
branches: [ main, master ]
8+
9+
jobs:
10+
demo:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- name: Setup Python
15+
uses: actions/setup-python@v4
16+
with:
17+
python-version: '3.11'
18+
- name: Run demo
19+
run: |
20+
python3 agential_demo/demo.py 2>&1 | tee demo.log
21+
- name: Upload demo log
22+
uses: actions/upload-artifact@v4
23+
with:
24+
name: demo-log
25+
path: demo.log
26+
27+
build_slides:
28+
needs: demo
29+
runs-on: ubuntu-latest
30+
steps:
31+
- uses: actions/checkout@v4
32+
- name: Install Rust toolchain (for tectonic)
33+
uses: actions-rs/toolchain@v1
34+
with:
35+
toolchain: stable
36+
profile: minimal
37+
override: true
38+
- name: Install tectonic
39+
run: cargo install tectonic --locked
40+
- name: Build slides PDF
41+
run: |
42+
cd agential_demo
43+
tectonic slides.tex
44+
- name: Upload slides PDF
45+
uses: actions/upload-artifact@v4
46+
with:
47+
name: slides-pdf
48+
path: agential_demo/slides.pdf

BugsInPy

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Subproject commit 11c5f1eea954a42132cfd06bf257766a7963e0fd

README.md

Lines changed: 75 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,16 @@ a3 scan . --triage github # uses GITHUB_TOKEN, free in CI
5252
a3 scan . --triage anthropic # uses ANTHROPIC_API_KEY
5353
```
5454

55+
#### Using GitHub Models locally (free, no API key needed)
56+
57+
If you have the [GitHub CLI](https://cli.github.com/) installed and authenticated (`gh auth login`), you can use GitHub Models for triage locally — no API key signup required:
58+
59+
```bash
60+
export GITHUB_TOKEN=$(gh auth token) && a3 scan . --triage github --verbose
61+
```
62+
63+
This exports your existing GitHub CLI token and uses it to access GitHub Models for agentic triage. In CI (GitHub Actions), `GITHUB_TOKEN` is already available automatically.
64+
5565
Or run scan and triage as separate steps:
5666

5767
```bash
@@ -199,6 +209,70 @@ a3 baseline diff --sarif results.sarif
199209
a3 baseline accept --sarif results.sarif
200210
```
201211

212+
### Two Usage Patterns
213+
214+
#### Scenario A: Whole-Repo Scan
215+
216+
Scan every Python file in one pass. Best for first-time adoption, weekly audits, or establishing a baseline.
217+
218+
```bash
219+
# Scan the entire repository
220+
a3 scan . --interprocedural --dse-verify --output-sarif results.sarif
221+
222+
# Agentic triage (optional — filters remaining false positives)
223+
a3 triage --sarif results.sarif --provider github --agentic \
224+
--output-sarif triaged.sarif
225+
226+
# Lock current findings as the baseline (ratchet start)
227+
a3 baseline accept --sarif triaged.sarif
228+
git add .a3-baseline.json && git commit -m "ci: establish a3 baseline" && git push
229+
```
230+
231+
The scheduled workflow (`a3-scheduled-scan.yml`) repeats this automatically every week.
232+
233+
#### Scenario B: Incremental — Auto-Invoke on Every Python File Change
234+
235+
After `a3 init . --copilot`, **no manual step is needed**. The generated `a3-pr-scan.yml` triggers whenever a `.py` file is added or modified:
236+
237+
```yaml
238+
on:
239+
push:
240+
branches: [main]
241+
paths: ["**.py"] # triggers ONLY when .py files change
242+
pull_request:
243+
branches: [main]
244+
paths: ["**.py"]
245+
```
246+
247+
**Example — adding a new file:**
248+
249+
```bash
250+
cat > src/payments.py << 'EOF'
251+
def charge(amount, discount):
252+
return amount / discount # DIV_ZERO: discount could be 0
253+
254+
def refund(amount, count):
255+
if count > 0:
256+
return amount / count # SAFE: guarded by count > 0
257+
return 0.0
258+
EOF
259+
260+
git add src/payments.py
261+
git commit -m "feat: add payment processing"
262+
git push # <-- CI auto-triggers, scans only src/payments.py
263+
```
264+
265+
What happens automatically:
266+
267+
1. Push triggers `a3-pr-scan.yml` (path filter matches `**.py`)
268+
2. Workflow detects changed files via `git diff --name-only`
269+
3. a3 scans only `src/payments.py` — finds DIV_ZERO in `charge()`, proves `refund()` safe
270+
4. Baseline diff: new bug not in baseline → **CI fails** until the bug is fixed or accepted
271+
272+
Non-Python changes (docs, configs, images) do not trigger the workflow at all.
273+
274+
---
275+
202276
### Using a different LLM provider (optional)
203277

204278
If you prefer Claude or GPT-5 via your own API key instead of GitHub Models:
@@ -291,7 +365,7 @@ a3 <target> [options] # legacy syntax, same behavior
291365
| `--output-sarif PATH` | Write SARIF 2.1.0 JSON |
292366
| `--triage [PROVIDER]` | Run agentic triage after scan (auto-detects API key, or specify: `openai`, `anthropic`, `github`) |
293367
| `--triage-model MODEL` | LLM model for integrated triage (default: provider-appropriate) |
294-
| `--save-results PATH` | Write results as pickle (default: `results/<name>_results.pkl`) |
368+
| `--save-results PATH` | Write results as JSON (default: `results/<name>_results.json`) |
295369
| `--verbose` | Detailed step-by-step output |
296370
| `--config PATH` | Path to `.a3.yml` |
297371
@@ -578,16 +652,6 @@ a3 baseline diff --sarif triaged.sarif --auto-issue
578652

579653
---
580654

581-
## Docker
582-
583-
```bash
584-
docker build -t a3 .
585-
docker run --rm -v $(pwd)/my_project:/target a3 /target
586-
docker run --rm -v $(pwd):/code a3 /code/myfile.py --functions
587-
```
588-
589-
---
590-
591655
## Architecture
592656

593657
```

a3_python/__init__.py

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,9 @@
1111
No heuristics. Grounded in Python→Z3 heap/transition/barrier model.
1212
"""
1313

14-
__version__ = "0.1.10"
14+
from importlib.metadata import version as _pkg_version, PackageNotFoundError
15+
16+
try:
17+
__version__: str = _pkg_version("a3-python")
18+
except PackageNotFoundError:
19+
__version__ = "0.0.0+dev"

a3_python/__main__.py

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
"""
2-
Allow running pyfromscratch as a module:
2+
Allow running a3_python as a module:
33
4-
python3.11 -m pyfromscratch <target> [options]
4+
python3 -m a3_python <target> [options]
55
6-
Delegates to pyfromscratch.cli:main().
6+
Delegates to a3_python.cli:main().
77
"""
88
import sys
9-
from .cli import main
9+
from .cli import _main_wrapper
1010

11-
sys.exit(main())
11+
sys.exit(_main_wrapper())

a3_python/barriers/context_aware_verification.py

Lines changed: 89 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -160,15 +160,24 @@ def verify_bug_with_full_context(
160160
# =====================================================================
161161
# LAYER 2: Synthesize barriers from preconditions
162162
# =====================================================================
163+
# CRITICAL FIX: Only mark safe if there is EVIDENCE that the barrier
164+
# condition holds (e.g., validated params, guard facts for this bug type).
165+
# Simply creating a barrier template is not proof — we need evidence
166+
# that the precondition (x != 0, len(x) > 0, x is not None) is enforced.
163167
if bug_variable:
164-
synthesized = self._synthesize_barrier_for_bug(
165-
bug_type, bug_variable, crash_summary
168+
# Check if there's evidence the precondition is satisfied
169+
has_evidence = self._has_precondition_evidence(
170+
bug_type, bug_variable, crash_summary, call_chain_summaries
166171
)
167-
if synthesized:
168-
result.synthesized_barriers.append(synthesized)
169-
result.is_safe = True
170-
result.verification_time_ms = (time.time() - start_time) * 1000
171-
return result
172+
if has_evidence:
173+
synthesized = self._synthesize_barrier_for_bug(
174+
bug_type, bug_variable, crash_summary
175+
)
176+
if synthesized:
177+
result.synthesized_barriers.append(synthesized)
178+
result.is_safe = True
179+
result.verification_time_ms = (time.time() - start_time) * 1000
180+
return result
172181

173182
# =====================================================================
174183
# LAYER 3: Learn invariants from codebase
@@ -187,7 +196,8 @@ def verify_bug_with_full_context(
187196
# LAYER 4: Interprocedural barrier propagation
188197
# =====================================================================
189198
interprocedural = self._propagate_barriers_interprocedurally(
190-
bug_type, bug_variable, call_chain_summaries
199+
bug_type, bug_variable, call_chain_summaries,
200+
crash_summary=crash_summary
191201
)
192202
if interprocedural:
193203
result.synthesized_barriers.extend(interprocedural)
@@ -432,6 +442,66 @@ def _check_learned_protection(
432442
# For now, conservative: return False
433443
return False
434444

445+
# =========================================================================
446+
# LAYER 2 SUPPORT: Precondition Evidence Check
447+
# =========================================================================
448+
449+
def _has_precondition_evidence(
450+
self,
451+
bug_type: str,
452+
bug_variable: Optional[str],
453+
crash_summary: CrashSummary,
454+
call_chain_summaries: List[CrashSummary],
455+
) -> bool:
456+
"""
457+
Check if there is concrete evidence that the barrier precondition holds.
458+
459+
A synthesized barrier is only trustworthy if there is evidence the code
460+
actually enforces the precondition. Without evidence, synthesizing a
461+
barrier template is meaningless — the code may not protect against the bug.
462+
463+
Evidence sources:
464+
- Guard facts that match the bug type (e.g., non-zero check for DIV_ZERO)
465+
- Validated params (caller checks value before passing)
466+
- Return guarantees from callees
467+
468+
Returns:
469+
True if there is concrete evidence the precondition is enforced
470+
"""
471+
from .guard_to_barrier import get_protected_bugs
472+
from ..semantics.interprocedural_guards import BUG_TYPE_TO_GUARD_TYPES
473+
474+
# 1. Check if crash summary has guard facts for this bug type
475+
relevant_guard_types = BUG_TYPE_TO_GUARD_TYPES.get(bug_type, set())
476+
477+
for block_id, guard_facts in crash_summary.intra_guard_facts.items():
478+
for guard_type, variable, extra in guard_facts:
479+
if guard_type in relevant_guard_types:
480+
# Found a guard for the right bug type
481+
# Check if it protects the right variable
482+
if bug_variable is None or variable is None:
483+
return True
484+
if bug_variable in str(variable) or str(variable) in bug_variable:
485+
return True
486+
487+
# 2. Check if the bug type is in guarded_bugs
488+
if bug_type in crash_summary.guarded_bugs:
489+
return True
490+
491+
# 3. Check validated params from call chain
492+
for summary in call_chain_summaries:
493+
for param_idx, validations in summary.validated_params.items():
494+
if validations and bug_variable and f'param_{param_idx}' == bug_variable:
495+
return True
496+
497+
# 4. Check return guarantees from callees
498+
for summary in call_chain_summaries:
499+
if bug_type in summary.guarded_bugs:
500+
return True
501+
502+
# No evidence found — the barrier can't be trusted
503+
return False
504+
435505
# =========================================================================
436506
# LAYER 4: Interprocedural Propagation
437507
# =========================================================================
@@ -440,18 +510,27 @@ def _propagate_barriers_interprocedurally(
440510
self,
441511
bug_type: str,
442512
bug_variable: Optional[str],
443-
call_chain_summaries: List[CrashSummary]
513+
call_chain_summaries: List[CrashSummary],
514+
crash_summary: Optional[CrashSummary] = None
444515
) -> List[BarrierCertificate]:
445516
"""
446517
Propagate barriers from callers to callees.
447518
448519
If caller validates parameter x, and callee uses x in a crash,
449520
the validation barrier protects the callee.
521+
522+
NOTE: A function's own return guarantees do NOT protect against its
523+
own internal bugs. E.g., if f() has return_guarantees={'nonnull'},
524+
that means f's *return value* is non-None, not that f is internally
525+
safe from NULL_PTR. Skip the crash function's own summary.
450526
"""
451527
propagated = []
452528

453-
# Check return guarantees from callees
529+
# Check return guarantees from callees (excluding crash function itself)
454530
for summary in call_chain_summaries:
531+
# A function's own return guarantee does not protect its internal ops
532+
if crash_summary is not None and summary is crash_summary:
533+
continue
455534
for guarantee_type in summary.return_guarantees:
456535
# Create barrier from guarantee
457536
if guarantee_type == 'nonempty' and bug_type == 'BOUNDS':

a3_python/barriers/dsos_sdsos.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1198,7 +1198,7 @@ class DSOSIntegrationConfig:
11981198

11991199
class DSOSSDSOSIntegration:
12001200
"""
1201-
Main integration class for DSOS/SDSOS in PythonFromScratch.
1201+
Main integration class for DSOS/SDSOS in a3-python.
12021202
12031203
Provides:
12041204
1. Problem analysis and method selection

a3_python/barriers/hscc2004.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
S. Prajna, A. Jadbabaie, G. J. Pappas.
66
"Safety verification of hybrid systems using barrier certificates." HSCC 2004.
77
8-
In PythonFromScratch, we use the same *barrier proof obligations* (Init/Unsafe/Step)
8+
In a3-python, we use the same *barrier proof obligations* (Init/Unsafe/Step)
99
but apply them to discrete transition systems extracted from Python bytecode.
1010
1111
This module provides a first practical, sound "hybrid-style" integration:

a3_python/barriers/lasserre_hierarchy.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1110,7 +1110,7 @@ def get_cached_counterexamples(self) -> List[List[float]]:
11101110

11111111
class LasserreIntegration:
11121112
"""
1113-
Main integration class for Lasserre hierarchy in PythonFromScratch.
1113+
Main integration class for Lasserre hierarchy in a3-python.
11141114
11151115
Provides the interface for the kitchen-sink orchestrator to use
11161116
Lasserre-based barrier synthesis with systematic degree lifting.

a3_python/barriers/parrilo_sos_sdp.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1181,7 +1181,7 @@ class SOSBarrierSynthesizer:
11811181
"""
11821182
Synthesizes barrier certificates using SOS-SDP.
11831183
1184-
This is the main integration point with the PythonFromScratch framework.
1184+
This is the main integration point with the a3-python framework.
11851185
It extracts polynomial models from program semantics and finds certificates
11861186
using Positivstellensatz-based reasoning.
11871187
"""
@@ -1596,7 +1596,7 @@ def operand_to_poly(operand) -> Optional[Polynomial]:
15961596

15971597
class ParriloSOSIntegration:
15981598
"""
1599-
Main integration class for Parrilo SOS-SDP in PythonFromScratch.
1599+
Main integration class for Parrilo SOS-SDP in a3-python.
16001600
16011601
This class provides the interface for the kitchen-sink orchestrator
16021602
to invoke SOS-based barrier synthesis and certification.

0 commit comments

Comments
 (0)