Skip to content

Commit 25e49e9

Browse files
ahhyoushhahhyoushh
committed
Merge commit '3614e1b8301dc66b0cddca70e582b3fc07bf6248' as 'wiki'
2 parents 0c37e16 + 3614e1b commit 25e49e9

73 files changed

Lines changed: 12024 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎wiki/.github/workflows/deploy.yml‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
name: Build and Deploy (Landing + Wiki)
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
8+
permissions:
9+
contents: write
10+
11+
jobs:
12+
build-and-deploy:
13+
runs-on: ubuntu-latest
14+
15+
steps:
16+
- name: Checkout
17+
uses: actions/checkout@v4
18+
19+
- name: Setup Ruby
20+
uses: ruby/setup-ruby@v1
21+
with:
22+
ruby-version: '3.1'
23+
bundler-cache: true
24+
25+
- name: Install Jekyll (remote theme)
26+
run: |
27+
cd wiki
28+
cat << 'EOF' > Gemfile
29+
source "https://rubygems.org"
30+
gem "github-pages", group: :jekyll_plugins
31+
gem "jekyll-remote-theme"
32+
EOF
33+
bundle install --jobs 4 --retry 3
34+
35+
- name: Build wiki
36+
run: |
37+
cd wiki
38+
bundle exec jekyll build -d _site
39+
40+
- name: Combine landing + wiki
41+
run: |
42+
mkdir public
43+
44+
# Copy landing to site root
45+
cp -R landing/* public/
46+
47+
# Put wiki under /wiki
48+
mkdir -p public/wiki
49+
cp -R wiki/_site/wiki/* public/wiki/
50+
51+
# Copy wiki theme assets to /assets
52+
mkdir -p public/assets
53+
cp -R wiki/_site/assets/* public/assets/
54+
55+
- name: Deploy to gh-pages
56+
uses: peaceiris/actions-gh-pages@v3
57+
with:
58+
github_token: ${{ secrets.GITHUB_TOKEN }}
59+
publish_branch: gh-pages
60+
publish_dir: public

‎wiki/.gitignore‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# ======================
2+
# Python / Environment
3+
# ======================
4+
__pycache__/
5+
*.py[cod]
6+
.Python
7+
env/
8+
venv/
9+
.venv/
10+
ENV/
11+
12+
# ======================
13+
# IDE / OS
14+
# ======================
15+
.vscode/
16+
.idea/
17+
*.swp
18+
*.swo
19+
*~
20+
.DS_Store
21+
Thumbs.db
22+
23+
# ======================
24+
# Logs (ABSOLUTE IGNORE)
25+
# ======================
26+
logs/
27+
logs/**
28+
scripts/logs/
29+
scripts/logs/**
30+
evaluation/logs/
31+
evaluation/logs/**
32+
33+
# Explicit safety
34+
*.log
35+
*.jsonl
36+
37+
# ======================
38+
# Generated / Large Files
39+
# ======================
40+
folder_embeddings.faiss
41+
folder_labels.json
42+
43+
evaluation/datasets/
44+
evaluation/datasets/**
45+
evaluation/datasets/**/*.pdf
46+
evaluation/datasets/**/*.txt
47+
evaluation/datasets/**/*.md
48+
49+
# ======================
50+
# Temp / Runtime
51+
# ======================
52+
files/
53+
sorted/
54+
temp/
55+
test_files/
56+
57+
# ======================
58+
# Backups
59+
# ======================
60+
*_backup.*
61+
*_old.*
62+
63+
# ======================
64+
# Wiki build artifacts
65+
# ======================
66+
wiki/_site/
67+
wiki/.jekyll-cache/
68+
wiki/.jekyll-metadata
69+
wiki/.sass-cache/
70+
71+
# ======================
72+
# Environment secrets
73+
# ======================
74+
.env
75+
76+
# ======================
77+
# Keep explicitly
78+
# ======================
79+
!README.md
80+
!wiki/**/*.md
81+
!evaluation/comprehensive_metrics.json

‎wiki/Complete_SFT_Handbook.pdf‎

12.9 KB
Binary file not shown.

‎wiki/FileSense_Launcher.bat‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
@echo off
2+
cd /d "%~dp0"
3+
call env\Scripts\activate.bat
4+
if errorlevel 1 (
5+
echo [!] Failed to activate virtual environment!
6+
echo Please ensure 'env' exists and is a valid virtualenv.
7+
pause
8+
exit /b
9+
)
10+
echo Starting FileSense Launcher...
11+
start "" pythonw scripts/launcher.py
12+
exit

‎wiki/FileSense_Launcher.sh‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/usr/bin/env bash
2+
set -e
3+
4+
cd "$(dirname "$0")"
5+
6+
echo "[*] Starting FileSense..."
7+
8+
PYTHON_CMD=""
9+
10+
if [[ -x "env/bin/python3" ]]; then
11+
PYTHON_CMD="env/bin/python3"
12+
echo "[+] Using virtualenv python"
13+
elif command -v python3 >/dev/null 2>&1; then
14+
PYTHON_CMD="python3"
15+
echo "[+] Using system python3"
16+
elif command -v python >/dev/null 2>&1; then
17+
PYTHON_CMD="python"
18+
echo "[+] Using system python"
19+
else
20+
echo "[!] Python not found"
21+
exit 1
22+
fi
23+
24+
exec "$PYTHON_CMD" scripts/launcher.py

‎wiki/LOGGING_ARCHITECTURE.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# Logging System Architecture
2+
3+
## Flow Diagram
4+
5+
```
6+
┌─────────────────────────────────────────────────────────────┐
7+
│ User runs script.py │
8+
│ python scripts/script.py --dir ./files │
9+
└────────────────────────┬────────────────────────────────────┘
10+
│
11+
▼
12+
┌───────────────────────────────┐
13+
│ Check --no-logs flag? │
14+
└───────┬───────────────┬───────┘
15+
│ │
16+
No │ │ Yes
17+
│ │
18+
▼ ▼
19+
┌────────────────────┐ ┌──────────────────┐
20+
│ setup_logger() │ │ Skip logging │
21+
│ Redirect stdout │ │ Run normally │
22+
└─────────┬──────────┘ └──────────────────┘
23+
│
24+
▼
25+
┌─────────────────────────────────────┐
26+
│ All print() calls captured to: │
27+
│ 1. Terminal (user sees output) │
28+
│ 2. Log buffer (stored in memory) │
29+
└─────────────────┬───────────────────┘
30+
│
31+
▼
32+
┌─────────────────────────────────────┐
33+
│ Script execution completes │
34+
│ (success or error) │
35+
└─────────────────┬───────────────────┘
36+
│
37+
▼
38+
┌────────────────────────────────┐
39+
│ Check --auto-save-logs flag? │
40+
└────────┬──────────────┬────────┘
41+
│ │
42+
No │ │ Yes
43+
│ │
44+
▼ ▼
45+
┌──────────────────────┐ ┌─────────────────────┐
46+
│ Prompt user: │ │ Auto-save with │
47+
│ "Save logs? (y/n)" │ │ timestamp │
48+
└──────┬───────────────┘ └──────────┬──────────┘
49+
│ │
50+
▼ │
51+
┌──────────────┐ │
52+
│ User says │ │
53+
│ 'y' or 'n'? │ │
54+
└──┬───────┬───┘ │
55+
│ │ │
56+
y │ │ n │
57+
│ │ │
58+
▼ ▼ │
59+
┌────┐ ┌────────┐ │
60+
│Save│ │Discard │ │
61+
└─┬──┘ └────────┘ │
62+
│ │
63+
▼ │
64+
┌──────────────────────┐ │
65+
│ Prompt for filename │ │
66+
│ (or use timestamp) │ │
67+
└──────────┬───────────┘ │
68+
│ │
69+
▼ ▼
70+
┌──────────────────────────────────────────┐
71+
│ Save to logs/run_YYYYMMDD_HHMMSS.log │
72+
│ or logs/custom_name.log │
73+
└──────────────────────────────────────────┘
74+
```
75+
76+
## Component Interaction
77+
78+
```
79+
┌─────────────────────────────────────────────────────────────┐
80+
│ script.py │
81+
│ ┌────────────────────────────────────────────────────┐ │
82+
│ │ Main Execution │ │
83+
│ │ • Parse arguments │ │
84+
│ │ • Setup logger (if enabled) │ │
85+
│ │ • Run file processing │ │
86+
│ │ • Handle log saving (finally block) │ │
87+
│ └────────────────────────────────────────────────────┘ │
88+
└───────────────────────┬─────────────────────────────────────┘
89+
│ imports
90+
▼
91+
┌─────────────────────────────────────────────────────────────┐
92+
│ logger.py │
93+
│ ┌────────────────────────────────────────────────────┐ │
94+
│ │ Logger Class │ │
95+
│ │ • write(message) → terminal + buffer │ │
96+
│ │ • get_logs() → return all buffered logs │ │
97+
│ │ • save_logs(filename) → write to file │ │
98+
│ │ • Thread-safe with locks │ │
99+
│ └────────────────────────────────────────────────────┘ │
100+
│ │
101+
│ ┌────────────────────────────────────────────────────┐ │
102+
│ │ Helper Functions │ │
103+
│ │ • setup_logger() → initialize & redirect stdout │ │
104+
│ │ • get_logger() → get global instance │ │
105+
│ │ • restore_stdout() → restore original stdout │ │
106+
│ └────────────────────────────────────────────────────┘ │
107+
└─────────────────────────────────────────────────────────────┘
108+
│
109+
▼
110+
┌─────────────────────────────────────────────────────────────┐
111+
│ File System │
112+
│ FileSense/ │
113+
│ └── logs/ │
114+
│ ├── run_20251205_143022.log │
115+
│ ├── run_20251205_145930.log │
116+
│ └── custom_name.log │
117+
└─────────────────────────────────────────────────────────────┘
118+
```
119+
120+
## Thread Safety
121+
122+
```
123+
┌─────────────────────────────────────────────────────────────┐
124+
│ Multi-threaded Processing │
125+
│ │
126+
│ Thread 1: process_file("doc1.pdf") │
127+
│ │ │
128+
│ └──► print("Processing doc1...") ──┐ │
129+
│ │ │
130+
│ Thread 2: process_file("doc2.pdf") │ │
131+
│ │ │ │
132+
│ └──► print("Processing doc2...") ──┤ │
133+
│ │ │
134+
│ Thread 3: process_file("doc3.pdf") │ │
135+
│ │ │ │
136+
│ └──► print("Processing doc3...") ──┤ │
137+
│ │ │
138+
│ ▼ │
139+
│ ┌──────────────────┐ │
140+
│ │ Logger.write() │ │
141+
│ │ with lock: │ │
142+
│ │ buffer.append │ │
143+
│ └──────────────────┘ │
144+
│ │ │
145+
│ ▼ │
146+
│ ┌──────────────────┐ │
147+
│ │ Synchronized │ │
148+
│ │ Log Buffer │ │
149+
│ └──────────────────┘ │
150+
└─────────────────────────────────────────────────────────────┘
151+
```
152+
153+
## Data Flow
154+
155+
```
156+
User Input (print statement)
157+
│
158+
▼
159+
sys.stdout.write() → Logger.write()
160+
│ │
161+
│ ├──► Terminal (immediate display)
162+
│ │
163+
│ └──► Log Buffer (with lock)
164+
│ │
165+
│ │ (stored in memory)
166+
│ │
167+
▼ ▼
168+
Script completes User prompted
169+
│ │
170+
│ ▼
171+
│ Save to file?
172+
│ │
173+
│ ▼
174+
└──────────────────────► logs/filename.log
175+
```

0 commit comments

Comments
 (0)