Implement CI workflow, add feature request template, and update project files

Author: totalbrain

.github/copilot-instructions.md

@@ -2,68 +2,52 @@
 Guidance for AI coding agents working on TokenOptimizer
 Keep this short, concrete, and code-focused. Update when `main.py` changes.
 -->
-
-# Copilot / Agent Instructions — TokenOptimizer
-
+# Copilot / Agent Instructions — TokenOptimizer (concise)
 Summary
-- This repository is a small single-file Python desktop app: `main.py` implements a Tkinter GUI and the token-optimization logic in the `AITokenCrusher` class.
-- Primary goal: conservative, offline token minimization for text/code using deterministic string and regex transforms.
-
-Where to look
-- Core app and logic: `main.py` (UI, options, `apply_optimizations`).
-- Project description / releases: `README.md`.
-- Static assets: `assets/` (images, screenshots).
-- Contribution workflow: `feature_request.md` describes feature requests.
-
-Big-picture architecture
-- Single-process GUI: `AITokenCrusher` constructs the UI and stores state in `self.options` (a dict of `tk.BooleanVar`) and `self.ui_elements`.
-- The optimization pipeline is implemented in `apply_optimizations(self, text)` and applied from `optimize()`.
-- UI <-> logic boundary: do not embed heavy business logic in widget callbacks; extend or refactor `apply_optimizations` when adding new transforms.
-
-Project-specific patterns & conventions
-- Option keys are snake_case strings in `self.options` (e.g. `remove_comments`, `single_line_mode`). Follow this pattern when adding flags.
-- Checkbuttons are created by iterating `self.options`; adding a new option requires:
-  1) add a `tk.BooleanVar` entry to `self.options` in `__init__`,
-  2) reference it in `apply_optimizations`, and
-  3) let the existing loop auto-create the checkbox.
-- UI state containers: use `self.ui_elements` to store references (avoid creating loose globals).
-- Theme handling: themes live in `self.themes` and switching occurs via `toggle_theme()` + `apply_theme()` — prefer updating these maps rather than hardcoding colors.
-
-Optimization details agents must preserve
-- `apply_optimizations` mixes regex and string replacements. Important examples to preserve exactly:
-  - Newline single-line mode uses the literal character `⏎` (replaces `\n`).
-  - Boolean/text replacements: `True`→`1`, `False`→`0`, `None`→`~`.
-  - Shortened keywords: `def `→`d `, `return `→`r `, `import `→`i `, `lambda `→`λ `.
-  - Operator replacements: `==`→`≡`, `!=`→`≠`, ` and `→`∧`, ` or `→`∨`.
-  - Short print: `print(` → `p(` (note the removed whitespace pattern).
-- These substitutions are intentionally non-standard/unicode; preserve their literal characters and avoid accidentally normalizing or escaping them.
-
-Run / build / debug
-- Run locally: `python main.py` (uses system Python with Tkinter). On Windows PowerShell:
+- Small, single-process Tkinter desktop app. Primary logic lives in `main.py` (`AITokenCrusher`) and an identical copy in `src/app.py`.
+- Purpose: deterministic, offline token minimization for text/code via string + regex transforms (keep substitutions literal).
+Where to look (high-value files)
+- `main.py` — canonical UI + optimization logic. Edit here for behavior changes.
+- `src/app.py` — duplicate of `main.py` (check both when refactoring).
+- `README.md` — user-facing description and release link.
+- `assets/` — screenshots and static images used in README.
+Big-picture architecture (quick)
+- UI and logic are co-located in `AITokenCrusher`:
+  - `self.options` (dict of `tk.BooleanVar`) drives which transforms run.
+  - `apply_optimizations(self, text)` contains the pipeline of regex/string transforms.
+  - `optimize()` reads input, calls `apply_optimizations`, updates output and stats.
+- The UI auto-creates checkboxes by iterating `self.options` — adding an option is three small steps (see below).
+Project-specific conventions (do these exactly)
+- Option lifecycle: add a `tk.BooleanVar` key in `self.options` (snake_case), implement the transform in `apply_optimizations`, and the checkbox appears automatically.
+- Keep UI references in `self.ui_elements` (avoid top-level globals).
+- Theme configuration lives in `self.themes` and is applied with `toggle_theme()` / `apply_theme()`; prefer updating theme maps over hardcoding colors.
+Critical transforms and literals to preserve
+- `single_line_mode` replaces newlines with the literal character `⏎` (not a word). Do not normalize or escape it.
+- Exact mappings (examples taken from `apply_optimizations`):
+  - Keywords: `def ` → `d `, `return ` → `r `, `import ` → `i `, `lambda ` → `λ `
+  - Booleans: `True` → `1`, `False` → `0`, `None` → `~`
+  - Operators: `==` → `≡`, `!=` → `≠`, ` and ` → `∧`, ` or ` → `∨`
+  - Print: `print(` → `p(`
+- Regex-based removals (comments, docstrings, type hints) are implemented with `re.sub` — preserve the intent and the approximate patterns when editing.
+Developer workflows / commands
+- Run app locally (PowerShell):
   ```pwsh
   python .\main.py
   ```
-- Packaging (not present in repo): releases show a Windows `.exe`. If creating builds, use a one-file GUI build (example):
+- Build a one-file Windows executable (only if requested):
   ```pwsh
   pyinstaller --onefile --windowed main.py
   ```
-  Only add packaging config if requested; keep runtime behaviour deterministic.
-
 Testing and verification
-- There are no automated tests. Changes that affect behavior require manual testing: launch the GUI, paste sample code, toggle options, and verify `apply_optimizations` output.
-- When adding new transforms, include small unit-testable helper functions and a minimal manual verification harness that calls `apply_optimizations(text)`.
-
-Safe editing guidance for agents
-- Be conservative: prefer adding small helper functions over large refactors.
-- Keep public names and option keys stable (changing keys requires UI and saved-state updates).
-- When introducing new dependencies, document why and prefer pure-Python stdlib solutions (this repo intentionally targets offline usage).
-
+- There are no automated tests. When changing transforms: add a small, importable helper function and unit tests under `tests/`.
+- Manual verification: run the GUI, paste input, toggle the relevant option(s), press *CRUSH TOKENS →*, and confirm output and the saved `%` metric.
+Small-change checklist for agents (use before committing)
+1. Add or rename an option: update `self.options` (tk.BooleanVar), add logic in `apply_optimizations`, keep key stable.
+2. Avoid changing existing substitution characters (⏎, ≡, ≠, ∧, ∨, λ, ~) unless maintainer OK.
+3. For UI changes, reuse `self.ui_elements` and the checkbox auto-creation loop.
+4. When refactoring, update both `main.py` and `src/app.py` (one is a copy).
 Files to reference in PRs
-- `main.py` — implementation and starting point for most changes.
-- `README.md` — user-facing description, check for mismatch with behaviour.
-- `assets/` — update assets if UI text or screenshots change.
-
-If something is unclear
-- Ask the maintainer which transforms are allowed to change semantics; otherwise assume existing substitutions must remain literal and reversible for humans.
-
-— End —
+- `main.py`, `src/app.py`, `README.md`, and `assets/`.
+If unclear
+- Ask the maintainer whether a transform may change semantics. Default: preserve existing mappings and prefer small, testable helpers.
+— End

.github/workflows/ci.yml

@@ -0,0 +1,23 @@
+name: CI - AI Token Crusher Tests
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+  schedule:
+    - cron: '0 0 * * *'  # Nightly at midnight UTC
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python 3.11
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install dependencies
+        run: pip install -r requirements.txt pytest
+      - name: Run tests
+        run: pytest tests/ -v

feature_request.md docs/feature_request.mdfeature_request.md renamed to docs/feature_request.md

@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"

pyproject.toml

@@ -0,0 +1,3 @@
+tkinter
+pytest
+pyinstaller

requirements.txt

@@ -0,0 +1,164 @@
+# main.py - AI Token Crusher v1.0
+import tkinter as tk
+from tkinter import ttk, filedialog, messagebox, scrolledtext
+import webbrowser
+import re
+
+class AITokenCrusher:
+    def __init__(self, root):
+        self.root = root
+        self.root.title("AI Token Crusher - Cut up to 75% tokens")
+        self.root.geometry("1280x820")
+        self.root.minsize(1000, 700)
+        self.root.configure(bg="#0d1117")
+
+        style = ttk.Style()
+        style.theme_use("clam")
+        style.configure("Title.TLabel", foreground="#58a6ff", font=("Segoe UI", 18, "bold"), background="#0d1117")
+        style.configure("TButton", padding=10, font=("Segoe UI", 9, "bold"))
+
+        self.options = {
+            "remove_comments": tk.BooleanVar(value=True),
+            "remove_docstrings": tk.BooleanVar(value=True),
+            "remove_blank_lines": tk.BooleanVar(value=True),
+            "remove_extra_spaces": tk.BooleanVar(value=True),
+            "single_line_mode": tk.BooleanVar(value=True),
+            "shorten_keywords": tk.BooleanVar(value=True),
+            "replace_booleans": tk.BooleanVar(value=True),
+            "use_short_operators": tk.BooleanVar(value=True),
+            "remove_type_hints": tk.BooleanVar(value=True),
+            "minify_structures": tk.BooleanVar(value=True),
+            "unicode_shortcuts": tk.BooleanVar(value=True),
+            "shorten_print": tk.BooleanVar(value=True),
+            "remove_asserts": tk.BooleanVar(value=True),
+            "remove_pass": tk.BooleanVar(value=True),
+        }
+
+        self.create_ui()
+
+    def create_ui(self):
+        main = tk.Frame(self.root, bg="#0d1117")
+        main.pack(fill="both", expand=True, padx=20, pady=20)
+
+        ttk.Label(main, text="AI Token Crusher", style="Title.TLabel").pack(pady=(0, 5))
+        ttk.Label(main, text="Cut up to 75% of tokens for Grok • GPT • Claude • Llama", 
+                 foreground="#8b949e", font=("Segoe UI", 11), background="#0d1117").pack(pady=(0, 20))
+
+        top = tk.Frame(main, bg="#0d1117")
+        top.pack(fill="both", expand=True)
+
+        input_frame = tk.LabelFrame(top, text=" Input Text / Code ", fg="#f0f6fc", bg="#161b22", font=("Segoe UI", 10, "bold"))
+        input_frame.pack(side="left", fill="both", expand=True, padx=(0, 10))
+        self.input_text = scrolledtext.ScrolledText(input_frame, font=("Consolas", 10), bg="#0d1117", fg="#c9d1d9")
+        self.input_text.pack(fill="both", expand=True, padx=10, pady=10)
+
+        btns = tk.Frame(input_frame, bg="#161b22")
+        btns.pack(pady=5)
+        ttk.Button(btns, text="Load File", command=self.load_file).pack(side="left", padx=5)
+        ttk.Button(btns, text="Copy Output", command=self.copy_output).pack(side="left", padx=5)
+
+        options_frame = tk.LabelFrame(top, text=" Optimization Techniques ", fg="#f0f6fc", bg="#161b22", font=("Segoe UI", 10, "bold"))
+        options_frame.pack(side="right", fill="y", padx=(10, 0))
+        canvas = tk.Canvas(options_frame, bg="#161b22", highlightthickness=0)
+        scrollbar = ttk.Scrollbar(options_frame, command=canvas.yview)
+        scroll_frame = tk.Frame(canvas, bg="#161b22")
+        canvas.create_window((0, 0), window=scroll_frame, anchor="nw")
+        canvas.configure(yscrollcommand=scrollbar.set)
+        canvas.pack(side="left", fill="both", expand=True, padx=10, pady=10)
+        scrollbar.pack(side="right", fill="y")
+
+        for key, var in self.options.items():
+            name = key.replace("_", " ").title().replace("Shorten", "Short").replace("Remove", "Strip")
+            tk.Checkbutton(scroll_frame, text=name, variable=var, bg="#161b22", fg="#c9d1d9", selectcolor="#21262d").pack(anchor="w", pady=2, padx=15)
+
+        ttk.Button(main, text="CRUSH TOKENS →", command=self.optimize).pack(pady=20)
+
+        output_frame = tk.LabelFrame(main, text=" Crushed Output (AI-Safe) ", fg="#f0f6fc", bg="#161b22", font=("Segoe UI", 10, "bold"))
+        output_frame.pack(fill="both", expand=True, pady=(10, 0))
+        self.output_text = scrolledtext.ScrolledText(output_frame, font=("Consolas", 10), bg="#0d1117", fg="#79c0ff")
+        self.output_text.pack(fill="both", expand=True, padx=10, pady=10)
+        ttk.Button(output_frame, text="Save Output", command=self.save_output).pack(pady=5)
+
+        self.stats = ttk.Label(main, text="Ready to crush tokens...", foreground="#79c0ff", font=("Consolas", 11, "bold"), background="#161b22")
+        self.stats.pack(pady=10)
+
+        footer = tk.Frame(main, bg="#0d1117")
+        footer.pack(pady=15)
+        links = [("GitHub", "https://github.com/totalbrain/TokenOptimizer"), ("Roadmap", "https://github.com/users/totalbrain/projects/1")]
+        for text, url in links:
+            link = tk.Label(footer, text=text, fg="#58a6ff", bg="#0d1117", cursor="hand2", font=("Segoe UI", 9, "underline"))
+            link.pack(side="left", padx=20)
+            link.bind("<Button-1>", lambda e, u=url: webbrowser.open(u))
+
+    def load_file(self):
+        path = filedialog.askopenfilename(filetypes=[("All Files", "*.*")])
+        if path:
+            with open(path, "r", encoding="utf-8") as f:
+                self.input_text.delete(1.0, tk.END)
+                self.input_text.insert(tk.END, f.read())
+
+    def copy_output(self):
+        output = self.output_text.get(1.0, tk.END).strip()
+        if output:
+            self.root.clipboard_clear()
+            self.root.clipboard_append(output)
+            messagebox.showinfo("Copied!", "Crushed text copied to clipboard!")
+
+    def save_output(self):
+        path = filedialog.asksaveasfilename(defaultextension=".txt")
+        if path:
+            with open(path, "w", encoding="utf-8") as f:
+                f.write(self.output_text.get(1.0, tk.END))
+            messagebox.showinfo("Saved", "Output saved successfully!")
+
+    def optimize(self):
+        text = self.input_text.get(1.0, tk.END).strip()
+        if not text:
+            messagebox.showwarning("Empty", "Paste or load text first!")
+            return
+
+        optimized = self.apply_optimizations(text)
+        self.output_text.delete(1.0, tk.END)
+        self.output_text.insert(tk.END, optimized)
+
+        before = len(text)
+        after = len(optimized)
+        saved = 100 * (before - after) / before if before else 0
+        self.stats.config(text=f"Before: {before:,} → After: {after:,} chars | Saved: {saved:.1f}%")
+
+    def apply_optimizations(self, text):
+        import re
+        if self.options["remove_comments"].get():
+            text = re.sub(r'#.*', '', text)
+            text = re.sub(r'"""[\s\S]*?"""|\'\'\'[\s\S]*?\'\'\'', '', text)
+        if self.options["remove_docstrings"].get():
+            text = re.sub(r'^[\r\n\s]*("""|\'\'\').*?\1', '', text, count=1, flags=re.DOTALL)
+        if self.options["remove_blank_lines"].get():
+            text = "\n".join(line for line in text.splitlines() if line.strip())
+        if self.options["remove_extra_spaces"].get():
+            text = re.sub(r'[ \t]+', ' ', text)
+        if self.options["single_line_mode"].get():
+            text = text.replace("\n", "⏎")
+        if self.options["shorten_keywords"].get():
+            rep = {"def ": "d ", "return ": "r ", "import ": "i ", "from ": "f ", "as ": "a ", "if ": "if", "class ": "c ", "lambda ": "λ "}
+            for k, v in rep.items(): text = text.replace(k, v)
+        if self.options["replace_booleans"].get():
+            text = text.replace("True", "1").replace("False", "0").replace("None", "~")
+        if self.options["use_short_operators"].get():
+            text = text.replace("==", "≡").replace("!=", "≠").replace(" and ", "∧").replace(" or ", "∨")
+        if self.options["remove_type_hints"].get():
+            text = re.sub(r':\s*[^=\n\->]+', '', text)
+            text = re.sub(r'->\s*[^:\n]+', '', text)
+        if self.options["minify_structures"].get():
+            text = re.sub(r',\s+', ',', text)
+            text = re.sub(r':\s+', ':', text)
+        if self.options["unicode_shortcuts"].get():
+            text = text.replace(" in ", "∈").replace(" not in ", "∉")
+        if self.options["shorten_print"].get():
+            text = re.sub(r'print\s*\(', 'p(', text)
+        return text.strip() + "\n"
+
+if __name__ == "__main__":
+    root = tk.Tk()
+    app = AITokenCrusher(root)
+    root.mainloop()