feat(gfql): WHERE clause with df_executor

CHANGELOG.md

@@ -8,6 +8,31 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 ## [Development]
 <!-- Do Not Erase This Section - Used for tracking unreleased changes -->
 
+### Added
+- **GFQL / WHERE** (experimental): Added `Chain.where` field for same-path WHERE clause constraints. New modules: `same_path_types.py`, `same_path_plan.py`, `df_executor.py` implementing Yannakakis-style semijoin reduction for efficient WHERE filtering. Supports equality, inequality, and comparison operators on named alias columns.
+- **GFQL / cuDF same-path**: Added execution-mode gate `GRAPHISTRY_CUDF_SAME_PATH_MODE` (auto/oracle/strict) for GFQL cuDF same-path executor. Auto falls back to oracle when GPU unavailable; strict requires cuDF or raises.
+- **Compute / hop**: Added `GRAPHISTRY_HOP_FAST_PATH` (set to `0`/`false`/`off`) to disable fast-path traversal for benchmarking or compatibility checks.
+
+### Performance
+- **Compute / hop**: Refactored hop traversal to precompute node predicate domains and unify direction handling; synthetic CPU benchmarks show modest median improvements with some regressions on undirected/range scenarios.
+- **GFQL / WHERE**: Use DF-native forward pruning for cuDF equality constraints to avoid host syncs (pandas path unchanged).
+- **Compute / hop**: Undirected traversal skips oriented-pair expansion when no destination filters; modest CPU gains in undirected benchmarks.
+- **Compute / hop**: Fast-path traversal uses domain-based visited/frontier tracking to avoid per-hop concat+dedupe overhead; modest CPU improvements in synthetic benchmarks.
+
+### Fixed
+- **GFQL / chain**: Fixed `from_json` to validate `where` field type before casting, preventing type errors on malformed input.
+- **GFQL / WHERE**: Fixed undirected edge handling in WHERE clause filtering to check both src→dst and dst→src directions.
+- **GFQL / WHERE**: Fixed multi-hop path edge retention to keep all edges in valid paths, not just terminal edges.
+- **GFQL / WHERE**: Fixed unfiltered start node handling with multi-hop edges in native path executor.
+
+### Infra
+- **GFQL / same_path**: Modular architecture for WHERE execution: `same_path_types.py` (types), `same_path_plan.py` (planning), `df_executor.py` (execution), plus `same_path/` submodules for BFS, edge semantics, multihop, post-pruning, and WHERE filtering.
+- **Benchmarks**: Added manual hop microbench + frontier sweep scripts under `benchmarks/` (not wired into CI).
+
+### Tests
+- **GFQL / df_executor**: Added comprehensive test suite (core, amplify, patterns, dimension) with 200+ tests covering Yannakakis semijoin, WHERE clause filtering, multi-hop paths, and pandas/cuDF parity.
+- **GFQL / cuDF same-path**: Added strict/auto mode coverage for cuDF executor fallback behavior.
+
 ## [0.50.4 - 2026-01-15]
 
 ### Fixed

ai/README.md

@@ -184,19 +184,38 @@ WITH_BUILD=0 WITH_TEST=0 ./test-cpu-local.sh
 
 ### GPU Testing - Fast (Reuse Base Image)
 
-Docker containers include: **pytest, mypy, ruff** (preinstalled)
+Docker containers include: **pytest, mypy, ruff, cudf** (preinstalled)
 
 ```bash
-# Reuse existing graphistry image (no rebuild)
-IMAGE="graphistry/graphistry-nvidia:${APP_BUILD_TAG:-latest}-${CUDA_SHORT_VERSION:-12.8}"
-
+# Container with cuDF available (cudf 25.10)
+IMAGE="graphistry/graphistry-nvidia:v2.50.0-13.0"
+
+# Run compute + GFQL tests with cuDF fallback (491 tests)
+# Uses CUDA_VISIBLE_DEVICES="" to avoid GPU driver issues
+docker run --rm -v /home/lmeyerov/Work/pygraphistry:/app -w /app \
+  -e CUDA_VISIBLE_DEVICES="" \
+  $IMAGE \
+  python -m pytest graphistry/tests/test_compute*.py tests/gfql/ref/ -q \
+    --ignore=tests/gfql/ref/test_ref_enumerator.py \
+    -k "not cudf_gpu_path"
+
+# Run GFQL ref tests only (372 tests)
+docker run --rm -v /home/lmeyerov/Work/pygraphistry:/app -w /app \
+  -e CUDA_VISIBLE_DEVICES="" \
+  $IMAGE \
+  python -m pytest tests/gfql/ref/ -q \
+    --ignore=tests/gfql/ref/test_ref_enumerator.py
+
+# With full GPU access (requires nvidia-container-toolkit)
 docker run --rm --gpus all \
-    -v "$(pwd):/workspace:ro" \
-    -w /workspace -e PYTHONPATH=/workspace \
-    $IMAGE pytest graphistry/tests/test_file.py -v
+    -v /home/lmeyerov/Work/pygraphistry:/app -w /app \
+    $IMAGE python -m pytest graphistry/tests/compute/ -q
 ```
 
-**Fast iteration**: Use this during development
+**Note**: Tests in `graphistry/tests/compute/predicates/` require real GPU access.
+Use `CUDA_VISIBLE_DEVICES=""` for cuDF import-path testing without GPU.
+
+**Fast iteration**: Use cuDF container during development
 **Full rebuild**: Use `./docker/test-gpu-local.sh` before merge
 
 ### Environment Control

benchmarks/README.md

@@ -0,0 +1,97 @@
+# Benchmarks
+
+Manual-only scripts for local performance checks. Not wired into CI.
+
+Summary results go into `benchmarks/RESULTS.md` (raw outputs stay in `plans/`).
+
+## Hop microbench
+
+Run a small set of hop() scenarios across synthetic graphs.
+
+```bash
+uv run python benchmarks/run_hop_microbench.py --runs 5 --output /tmp/hop-microbench.md
+```
+
+## Frontier sweep
+
+Sweep seed sizes on a fixed linear graph.
+
+```bash
+uv run python benchmarks/run_hop_frontier_sweep.py --runs 5 --nodes 100000 --edges 200000 --output /tmp/hop-frontier.md
+```
+
+Notes:
+- Use `--engine cudf` for GPU runs when cuDF is available.
+- Scripts print a table to stdout; `--output` writes Markdown results.
+
+## Chain vs Yannakakis
+
+Compare regular `chain()` against the Yannakakis same-path executor on synthetic graphs.
+
+```bash
+uv run python benchmarks/run_chain_vs_samepath.py --runs 7 --warmup 1 --output /tmp/chain-vs-samepath.md
+```
+
+To toggle non-adjacent WHERE experiments on synthetic scenarios:
+
+```bash
+uv run python benchmarks/run_chain_vs_samepath.py \
+  --non-adj-mode value_prefilter \
+  --non-adj-value-card-max 500 \
+  --non-adj-order selectivity \
+  --non-adj-bounds \
+  --runs 7 --warmup 1
+```
+
+## Real-data GFQL
+
+Run GFQL chain scenarios on demo datasets plus WHERE scenarios (df_executor), with separate sections and a per-section score.
+
+```bash
+uv run python benchmarks/run_realdata_benchmarks.py --runs 7 --warmup 1 --output /tmp/realdata-gfql.md
+```
+
+To test categorical domains for redteam:
+
+```bash
+uv run python benchmarks/run_realdata_benchmarks.py --datasets redteam50k --redteam-domain-categorical --runs 9 --warmup 2
+```
+
+To experiment with non-adjacent WHERE modes:
+
+```bash
+uv run python benchmarks/run_realdata_benchmarks.py \
+  --datasets redteam50k \
+  --non-adj-mode value_prefilter \
+  --non-adj-value-card-max 500 \
+  --non-adj-order selectivity \
+  --non-adj-bounds \
+  --runs 7 --warmup 1
+```
+
+To enable OpenTelemetry spans for df_executor:
+
+```bash
+GRAPHISTRY_OTEL=1 \
+GRAPHISTRY_OTEL_DETAIL=1 \
+uv run --with opentelemetry-api --with opentelemetry-sdk \
+  python benchmarks/run_realdata_benchmarks.py --datasets redteam50k --runs 3 --warmup 1
+```
+
+To export spans to OTLP (optional):
+
+```bash
+GRAPHISTRY_OTEL=1 \
+GRAPHISTRY_OTEL_EXPORTER=otlp \
+OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
+uv run --with opentelemetry-api --with opentelemetry-sdk --with opentelemetry-exporter-otlp \
+  python benchmarks/run_realdata_benchmarks.py --datasets redteam50k --runs 3 --warmup 1
+```
+
+To limit datasets:
+
+```bash
+uv run python benchmarks/run_realdata_benchmarks.py --datasets redteam50k,transactions --runs 7 --warmup 1
+```
+
+Available datasets: `redteam50k`, `transactions`, `facebook_combined`, `honeypot`, `twitter_demo`, `lesmiserables`, `twitter_congress`, `all`.

benchmarks/RESULTS.md

@@ -0,0 +1,16 @@
+# Benchmark Results Log
+
+Summary-only log for notable benchmark runs. Raw per-scenario outputs live in
+`plans/` (gitignored) and should be referenced here.
+
+| Date | Commit | Scripts | Summary | Notes |
+|------|--------|---------|---------|-------|
+| 2026-01-17 | f492135e (feat/where-clause-executor) | `run_chain_vs_samepath.py` (median-of-7, warmup-1); `run_realdata_benchmarks.py` (median-of-7, warmup-1) | Synthetic: yann/regular median ~0.51x (52/54 wins). Real data: expanded to 7 datasets, medians ~30–173ms. | Raw outputs: `plans/pr-886-where/benchmarks/phase-12-revert-8-11.md`, `plans/pr-886-where/benchmarks/phase-13-realdata.md` |
+| 2026-01-17 | 7080e356 (feat/where-clause-executor) | `run_realdata_benchmarks.py` (median-of-7, warmup-1) | Real data now includes WHERE (df_executor): redteam ~14s, transactions ~11s, others ~14–282ms. Chain-only medians ~31–175ms. | Raw outputs: `plans/pr-886-where/benchmarks/phase-14-realdata.md` |
+| 2026-01-17 | 2e2e7e18 (feat/where-clause-executor) | `run_realdata_benchmarks.py` (median-of-7, warmup-1) | Added per-section scores. Chain score (median of medians) 72.78ms; WHERE score 247.07ms. | Raw outputs: `plans/pr-886-where/benchmarks/phase-14-realdata.md` |
+| 2026-01-17 | 6bec468b (feat/where-clause-executor) | `run_realdata_benchmarks.py --datasets redteam50k --runs 9 --warmup 2` | Redteam-only rerun: chain score 157.83ms; WHERE score 13.12s. Low selectivity (WHERE keeps ~83.6% nodes / 74.3% edges). | Raw outputs: `plans/pr-886-where/benchmarks/phase-14-redteam-highruns.md`, `plans/pr-886-where/benchmarks/phase-14-redteam-selectivity.md` |
+| 2026-01-17 | 6bec468b (feat/where-clause-executor) | `run_realdata_benchmarks.py --datasets redteam50k --redteam-domain-categorical --runs 9 --warmup 2` | Redteam categorical domains: chain score 164.63ms; WHERE score 13.12s (no meaningful change). | Raw outputs: `plans/pr-886-where/benchmarks/phase-14-redteam-cat.md` |
+| 2026-01-18 | 20aab655 (feat/where-clause-executor) | `run_realdata_benchmarks.py --datasets redteam50k` (median-of-7, warmup-1) with `GRAPHISTRY_HOP_FAST_PATH=0/1` | Fast path on is slower for chain (~6-13%, score 164.89ms vs 154.75ms); WHERE delta likely noise (12.07s vs 13.12s). | Raw outputs: `plans/pr-886-where/benchmarks/phase-17-redteam-fastpath-off.md`, `plans/pr-886-where/benchmarks/phase-17-redteam-fastpath-on.md` |
+| 2026-01-18 | 7e3da877 (feat/where-clause-executor) | `run_realdata_benchmarks.py --datasets redteam50k` (median-of-7, warmup-1) with baseline vs `--non-adj-mode value_prefilter --non-adj-value-card-max 500 --non-adj-order selectivity --non-adj-bounds` | Non-adj value+prefilter dropped redteam WHERE from 12.96s → 0.35s; needs parity validation. Chain-only roughly unchanged. | Raw outputs: `plans/pr-886-where/benchmarks/phase-18-redteam-baseline.md`, `plans/pr-886-where/benchmarks/phase-18-redteam-value_prefilter.md` |
+| 2026-01-18 | 7e3da877 (feat/where-clause-executor) | `run_realdata_benchmarks.py --datasets redteam50k,transactions,facebook_combined` (median-of-7, warmup-1) baseline vs `--non-adj-mode value_prefilter --non-adj-value-card-max 500 --non-adj-order selectivity --non-adj-bounds` | WHERE: redteam 11.1s → 0.33s, transactions ~10.0s → ~10.1s, facebook ~239ms → ~244ms. | Raw outputs: `plans/pr-886-where/benchmarks/phase-18-realdata-baseline.md`, `plans/pr-886-where/benchmarks/phase-18-realdata-value_prefilter.md` |
+| 2026-01-18 | 7e3da877 (feat/where-clause-executor) | `run_chain_vs_samepath.py` (median-of-7, warmup-1) baseline vs `--non-adj-mode value_prefilter --non-adj-value-card-max 500 --non-adj-order selectivity --non-adj-bounds` | Synthetic: small deltas; dense non-adj still slower than regular. | Raw outputs: `plans/pr-886-where/benchmarks/phase-18-synth-baseline.md`, `plans/pr-886-where/benchmarks/phase-18-synth-value_prefilter.md` |

benchmarks/otel_setup.py

@@ -0,0 +1,66 @@
+"""Optional OpenTelemetry setup for benchmarks.
+
+This keeps deps optional: if opentelemetry is missing, it no-ops.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from typing import Optional
+
+
+def setup_tracer() -> bool:
+    if os.environ.get("GRAPHISTRY_OTEL", "").strip().lower() not in {"1", "true", "yes", "on"}:
+        return False
+
+    try:
+        from opentelemetry import trace  # type: ignore
+        from opentelemetry.sdk.trace import TracerProvider  # type: ignore
+        from opentelemetry.sdk.trace.export import (  # type: ignore
+            BatchSpanProcessor,
+            ConsoleSpanExporter,
+            SimpleSpanProcessor,
+        )
+        from opentelemetry.sdk.resources import Resource  # type: ignore
+    except Exception:
+        print("OpenTelemetry SDK not installed; spans will not be exported.", file=sys.stderr)
+        return False
+
+    exporter_kind = os.environ.get("GRAPHISTRY_OTEL_EXPORTER", "console").strip().lower()
+    processor = None
+
+    if exporter_kind == "otlp":
+        exporter = _make_otlp_exporter()
+        if exporter is None:
+            return False
+        processor = BatchSpanProcessor(exporter)
+    else:
+        processor = SimpleSpanProcessor(ConsoleSpanExporter())
+
+    provider = trace.get_tracer_provider()
+    if not hasattr(provider, "add_span_processor"):
+        service_name = os.environ.get("OTEL_SERVICE_NAME", "graphistry")
+        provider = TracerProvider(resource=Resource.create({"service.name": service_name}))
+        trace.set_tracer_provider(provider)
+
+    provider.add_span_processor(processor)
+    return True
+
+
+def _make_otlp_exporter() -> Optional[object]:
+    endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "").strip()
+    try:
+        from opentelemetry.exporter.otlp.proto.http.trace_exporter import (  # type: ignore
+            OTLPSpanExporter,
+        )
+        return OTLPSpanExporter(endpoint=endpoint or None)
+    except Exception:
+        try:
+            from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (  # type: ignore
+                OTLPSpanExporter,
+            )
+            return OTLPSpanExporter(endpoint=endpoint or None)
+        except Exception:
+            print("OTLP exporter not available; install opentelemetry-exporter-otlp.", file=sys.stderr)
+            return None