diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 989dd8d..ebd6ff1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,11 +1,24 @@ # CONTRIBUTING This document contains the list of issues, suggestions and improvements that can be added to this project and a proper guide on how to contribute to the project. +- [CONTRIBUTING](#contributing) + - [Registration](#registration) + - [Guildlines :](#guildlines-) + - [List of major issues :](#list-of-major-issues-) + - [List of Improvements and Suggestions](#list-of-improvements-and-suggestions) + - [Improvments to the UI](#improvments-to-the-ui) + - [Long-term Goals :](#long-term-goals-) + - [Improvements to Backend :-](#improvements-to-backend--) + - [Long term Goals :](#long-term-goals--1) + - [In game control :](#in-game-control-) + - [Cross platform compatibility :](#cross-platform-compatibility-) + - [Other Known Bugs and Issues : 🪲](#other-known-bugs-and-issues--) ## Registration -To be eligible for MLSA X HACKTOBERFEST: -[Star this repo](https://github.com/keploy/keploy) -[Register here for MLSAKKIIT](https://register.mlsakiit.com/) -[Register on HacktoberFest](https://hacktoberfest.com/auth/) +>[!important] Attention +All contributors must do the following to be eligible for MLSA X HACKTOBERFEST: +- [Star this repository](https://github.com/keploy/keploy) +- [Register here on MLSA KIIT website](https://register.mlsakiit.com/) +- [Register on HacktoberFest Official Website](https://hacktoberfest.com/auth/) ## Guildlines : Adhere to [Hacktober Fest Guidlines](https://hacktoberfest.com/) and maintain common ettiquette of contributing to an open source project. @@ -13,57 +26,66 @@ Adhere to [Hacktober Fest Guidlines](https://hacktoberfest.com/) and maintain co If you have any questions regarding contributing to this repository please contact the contributor. +### List of major issues : +>[!IMPORTANT] +It is highly recommended for contributors to first have a look at the list of major issues work on them with higher priority. -## UI Improvements -1. Add markdown support for gemini's reponses. -2. Improve the chat window UI such that it is easier to tell apart the messages of the user and Pixly. -3. Add the ability to view images results from the web and play recommended youtube videos directly within the overlay. -4. Make the window resizable and all window elements scalable. -5. Add window control buttons for minimise and fullscreen (to be added after the previous implementation) +**Related to UI :-** +1. Make the window resizable and all window elements scalable. +2. Add window control buttons for minimise and fullscreen (to be added after the previous implementation) +3. Whenever you hover over the screenshot button, placehoder text is inserted in the chatbox but it doesn't go away after you stop hovering, and you have to manually press backspace to remove the text. +4. Refactor the codebase of the overlay modular so that it is easier to work with. + +**Frontend+Backend :-** +1. Chat history is not stored, if you try to follow up gemini with what you asked in the previous chat it will have 0 idea what you are talking about. -### Long-term Goals : -6. Add the ability for users to add custom themes of the overlay, different themes belonging to different games, so when the overlay can automatically apply a certain theme when a particular game is detected. +>[!TIP] Here is a suggested solution: +Store the chats of the user in a database, (we are already using sqlite for screenshots, might as well use it), then when we give a new prompt to gemini, old chats are added to the prompt. +- Only store the last 30 chats or so. +- Every game will have its own database table, i.e. chats are stored on a per game basis. +- Additional meta-data such as timestamp should also be stored so when user asked "What did I do yesterday?" it should be able to retrieve the screenshot from 24 hours ago etc. -## Main Improvements -1. Chat history is not stored, add the ability to store the chat history the user across various sessions, preferably in a per game basis, in the vector database. ->[!Important] Also add the ability to view the stored chats across various sessions/games from the overlay, add the configuration in the settings menu to delete and edit them. Add a setting to set how many chats per game/session to store. - -2. Make a better system for sending existing screenshots. Improve the keyword search and allow user to edit the prompt before sending the screenshot instead of using the default prompt. Allow the chatbot agent to automatically pull a screenshot based on a given time and date duration. - -3. Add more entries about wikis, guides, youtube videos, forum posts about more games, especially single player story based titles like `Elden Ring, Hollow Knight : Silksong, Black Myth: Wukong,Cyberpunk 2077` - -4. Add cross platform support, (change the win32 dependency to an alternative) - -5. Improve the project structure to better align with [best practices](https://github.com/zhanymkanov/fastapi-best-practices) when using FastAPI. The project already defines pydantic schemas. Also Example project structure. : -``` -backend/ -│ -├── main.py # Entry point (FastAPI app) -│ -├── routers/ # Route definitions (API endpoints) -│ ├── chat.py -│ ├── screenshots.py -│ ├── game_detection.py -│ └── knowledge.py -│ -├── schemas/ # Pydantic models -│ ├── chat.py -│ ├── screenshot.py -│ ├── game_detection.py -│ └── knowledge.py -│ -├── services/ # Business logic / helper functions -│ ├── chatbot.py -│ ├── screenshot_service.py -│ ├── knowledge_service.py -│ └── vector_service.py -│ -└── core/ - ├── config.py # Settings, env vars, constants - └── logger.py # Central logging setup - -``` +>[!Important] UI/UX Addition : +- Add the ability to view the stored chats across various sessions/games from the overlay. +- Add the configuration in the settings menu to read, delete and edit them. +- Add a setting to set how many chats per game/session to store. + +**Backend :-** +1. Chroma db vector collections aren't searched properly, this may have to do with the chroma client not being initialised properly or the collections are not being created properly in `get_or_create_collection()` or the incorrect implementation of `search_knowledge()` in *vector_service.py*. This issue requires a more thorough investigation. +2. Web Scrapper in *knowledge_manager.py* sometimes gets blocked by certain websites, (*namely* the ones present in *minecraft.csv*) +>[!tip] Recommended Solutions : +- Use Proxies to circumvent IP bans. +- Rotate a list of User Agents and headers. +- Make it asynchronous using `asyncio + httpx` + + + + +## List of Improvements and Suggestions +>[!tip] Feel free to give us any of your ideas, suggestions and feedback to add to this list. + +### Improvments to the UI +1. Add Markdown support for Gemini's reponses. +2. Improve the chat Window UI such that it is easier to tell apart the messages of the user and Pixly. +3. Add the ability to view images results from the web and play recommended youtube videos directly within the overlay. + + + +#### Long-term Goals : +1. Add the ability for users to add custom themes of the overlay, different themes belonging to different games, so when the overlay can automatically apply a certain theme when a particular game is detected. + +### Improvements to Backend :- + +1. Add more .csv entries about wikis, guides, youtube videos, forum posts about more games, especially single player story based titles like `Elden Ring, Hollow Knight : Silksong, Black Myth: Wukong,Cyberpunk 2077` +2. Implement a Better way to store screenshots: +>[!tip] Suggested Improvemments : +- Vectorise the screenshots as well. +- Add tool calling for the agent to call a tool to retrieve the screenshot from a specific time or from a specific game. +1. Improve the Web Scrapper : + +In Addition to fixing the above issues, add the ability to scrape youtube audio transcriptions. + ### Long term Goals : #### In game control : @@ -73,9 +95,10 @@ Add the ability for the agent to control the game and play the game for you and Reliance on the win32 api for taking screenshots means we can't transition to a different platform, and are stuck with Windows for now. In the future we may wanna add cross platform compatibility with Linux. -## Known Bugs and Issues : 🪲 +## Other Known Bugs and Issues : 🪲 +> [!important] These are a bit obscure and their causes aren't known yet. +1. Overlay hangs and then crashes when turning off the *enable screenshots setting.* +2. `game_detection.py`, it reports the incorrect game being detected, in some cases. +3. After adding your API Key from the overlay, sometimes it shows that the user has added their key, sometimes it doesn't. -1. Overlay hangs and then crashes when turning off the enable screenshots setting. -2. Diagnose `knowledge_manager.py`, it is unable to scrape the given webpages. Returns a `403: Forbidden Error`, Possibly has to do with anti-bot anti-web_scraping measures. -3. Diagnose `game_detection.py`, sometimes it always reports the current game being played as minecraft. diff --git a/README.md b/README.md index 1c677f2..bd6cf5b 100644 --- a/README.md +++ b/README.md @@ -14,13 +14,15 @@ Pixly - Your AI Gaming Assistant 🎮 Pixly is a desktop overlay that acts as your gaming assistant, combining AI chat with automated, privacy-friendly screenshot capture and a game-specific Retrieval-Augmented Generation (RAG) knowledge base. Pixly detects what game you're playing, retrieves relevant, curated knowledge (wikis, user-supplied YouTube descriptions, and forum posts) via a local vector database, and grounds Gemini responses on those sources. -**🎃 Hacktoberfest 2025 Participant** | Join us in making gaming more accessible with AI! Also make sure to [star this repo](https://github.com/keploy/keploy). +Make sure to star our repository, your support is much appreciated. + +>[!important] 🎃 Hacktoberfest 2025 Participant +Please make sure to [star this repo](https://github.com/keploy/keploy). ## 📋 Table of Contents - [📋 Table of Contents](#-table-of-contents) -- [🎮 What Pixly Does](#-what-pixly-does) - [🤝 Contributing, Setup and Install](#-contributing-setup-and-install) - - [🎃 Hacktoberfest 2025 - How to Contribute](#-hacktoberfest-2025---how-to-contribute) +- [🎮 What Pixly Does](#-what-pixly-does) - [🏗️ Architecture Overview](#️-architecture-overview) - [1) UI Overlay (`overlay.py`)](#1-ui-overlay-overlaypy) - [2) Backend API (`backend/`)](#2-backend-api-backend) @@ -35,14 +37,6 @@ Pixly is a desktop overlay that acts as your gaming assistant, combining AI chat - [📄 License](#-license) - [🙏 Acknowledgments](#-acknowledgments) -## 🎮 What Pixly Does - -- 🤖 Intelligent, game-focused chat using Google Gemini with a "Game Expert" system prompt -- 🎯 Contextual help based on your active game (process detection and/or user message) -- 📸 Optional screenshot-powered context for visual analysis -- 🔍 RAG pipeline over per-game CSV knowledge with local vector search (Chroma) -- 💻 Modern desktop overlay for chatting, settings, and screenshot gallery - ## 🤝 Contributing, Setup and Install **We welcome Hacktoberfest 2025 contributors!** Whether you're adding new games to the knowledge base, improving the UI, or enhancing AI capabilities, your contributions matter. @@ -50,17 +44,15 @@ Pixly is a desktop overlay that acts as your gaming assistant, combining AI chat - 📖 For Contributing Visit [CONTRIBUTING.md](https://github.com/MLSAKIIT/pixly/blob/main/CONTRIBUTING.md) - ⚙️ For Setup and Installation visit [INSTALL.md](https://github.com/MLSAKIIT/pixly/blob/main/INSTALL.md) -### 🎃 Hacktoberfest 2025 - How to Contribute -This project is participating in **Hacktoberfest 2025**! Here are some ways you can contribute: +## 🎮 What Pixly Does -1. **Add Game Knowledge**: Contribute CSV files with game wikis, YouTube videos, and forum links -2. **Improve Documentation**: Enhance README, add tutorials, or create guides -3. **Fix Bugs**: Check our issues and help resolve reported bugs -4. **Add Features**: Implement new features like support for more games or UI improvements -5. **Optimize Performance**: Improve RAG retrieval, vector search, or screenshot handling +- 🤖 Intelligent, game-focused chat using Google Gemini with a "Game Expert" system prompt +- 🎯 Contextual help based on your active game (process detection and/or user message) +- 📸 Optional screenshot-powered context for visual analysis +- 🔍 RAG pipeline over per-game CSV knowledge with local vector search (Chroma) +- 💻 Modern desktop overlay for chatting, settings, and screenshot gallery -**Note**: Make sure to follow our [Code of Conduct](CODE_OF_CONDUCT.md) and contribution guidelines! ## 🏗️ Architecture Overview @@ -150,20 +142,29 @@ The detection result is passed into the RAG layer to scope retrieval to the acti ## 📁 Project Structure ``` -Hacktober Fest/ -├── backend/ -│ ├── __init__.py # FastAPI app initialization -│ ├── backend.py # API endpoints and routing +pixly/ +├── backend/ +│ ├── backend.py # FastAPI app initialization +├── routers/ # Contains all the API Routers +| ├── chat.py # Stores chat endpoints +| ├── game_detection.py # Stores game detection and vector search endpoints +| ├── screenshot.py # Stores screenshot endpoints +| ├── setting.py # Stores settings endpoints +├── services/ # Contains all the backend services. │ ├── chatbot.py # Gemini integration, RAG-aware chat, runtime reconfigure │ ├── screenshot.py # Encrypted screenshot capture, DB ops, delete support │ ├── game_detection.py # Process/message/screenshot-based game detection │ ├── knowledge_manager.py # CSV ingestion and content extraction (wiki/forum) │ └── vector_service.py # Chroma collections, embeddings, and search +├── schemas/ # Contains the schemas for the various requests +| ├── chat.py +| ├── game_detection.py +| ├── knowledge_search.py +| ├── settings.py ├── overlay.py # CustomTkinter overlay (chat, settings, screenshot viewer) ├── games_info/ # Per-game CSVs (e.g., minecraft.csv) ├── vector_db/ # Chroma persistent storage ├── PROMPTS.txt # System persona + RAG grounding instructions -├── test_system.py # Local API test harness ├── run.py # Backend server launcher ├── pyproject.toml # Dependencies and metadata ├── screenshots.db # Encrypted screenshot database (auto-created) diff --git a/backend/__init__.py b/backend/__init__.py index d593ada..e69de29 100644 --- a/backend/__init__.py +++ b/backend/__init__.py @@ -1,2 +0,0 @@ -from fastapi import FastAPI -app = FastAPI() \ No newline at end of file diff --git a/backend/backend.py b/backend/backend.py index f8ece39..0002845 100644 --- a/backend/backend.py +++ b/backend/backend.py @@ -1,241 +1,10 @@ -"""All backend endpoints exist here""" -from fastapi import FastAPI, HTTPException -import os -import uvicorn -from .chatbot import chat_with_gemini, set_api_key -from .screenshot import start_screenshot_capture, stop_screenshot_capture, get_recent_screenshots, get_screenshot_by_id, get_screenshot_stats, delete_screenshot -from .game_detection import detect_current_game, get_available_games as get_detection_games -from .knowledge_manager import get_available_games as get_csv_games, validate_csv_structure -from .vector_service import add_game_knowledge, search_knowledge, get_game_stats, list_available_games -from pydantic import BaseModel -from typing import Optional, List -from . import app +"""Backend Server Exists here""" +from fastapi import FastAPI +from routers import chat, screenshot, game_detection, settings -class ChatMessage(BaseModel): - message: str - image_data: Optional[str] = None # Base64 encoded image data +app = FastAPI() -class GameDetectionRequest(BaseModel): - message: Optional[str] = None - -class KnowledgeSearchRequest(BaseModel): - query: str - content_types: Optional[List[str]] = None - limit: Optional[int] = 5 - -class ApiKeyRequest(BaseModel): - api_key: str - -@app.post("/chat") -async def chat(message: ChatMessage): - return await chat_with_gemini(message.message, message.image_data) - -# Dummy functions to test functionality,, carry no use -@app.get("/taskA") -def run_task_a(): - return {"status": "ok", "message": "Task A executed"} - -@app.get("/taskB") -def run_task_b(): - return {"status": "ok", "message": "Task B executed"} - -# Screenshot endpoints -@app.post("/screenshots/start") -def start_screenshots(interval: int = 30): - """Start automatic screenshot capture.""" - start_screenshot_capture(interval) - return {"status": "ok", "message": f"Screenshot capture started with {interval}s interval"} - -@app.post("/screenshots/stop") -def stop_screenshots(): - """Stop automatic screenshot capture.""" - stop_screenshot_capture() - return {"status": "ok", "message": "Screenshot capture stopped"} - -@app.get("/screenshots/recent") -def get_recent_screenshots_endpoint(limit: int = 10, application: str = None): - """Get recent screenshots.""" - screenshots = get_recent_screenshots(limit=limit, application=application) - return {"status": "ok", "screenshots": screenshots} - -@app.get("/screenshots/stats") -def get_screenshot_stats_endpoint(): - """Get screenshot statistics.""" - stats = get_screenshot_stats() - return {"status": "ok", "stats": stats} - -@app.get("/screenshots/{screenshot_id}") -def get_screenshot_endpoint(screenshot_id: int): - """Get screenshot data by ID.""" - screenshot_data = get_screenshot_by_id(screenshot_id) - if screenshot_data: - import base64 - return {"status": "ok", "data": base64.b64encode(screenshot_data).decode('utf-8')} - else: - return {"status": "error", "message": "Screenshot not found"} - -@app.delete("/screenshots/{screenshot_id}") -def delete_screenshot_endpoint(screenshot_id: int): - """Delete screenshot by ID.""" - try: - deleted = delete_screenshot(screenshot_id) - if deleted: - return {"status": "ok", "message": f"Deleted screenshot {screenshot_id}"} - else: - raise HTTPException(status_code=404, detail="Screenshot not found") - except HTTPException: - raise - except Exception as e: - raise HTTPException(status_code=500, detail=f"Error deleting screenshot: {str(e)}") - -# Game Detection endpoints -@app.post("/games/detect") -def detect_game(request: GameDetectionRequest): - """Detect current game from process/screenshot/message.""" - try: - detected_game = detect_current_game(request.message) - return { - "status": "ok", - "detected_game": detected_game, - "message": f"Detected game: {detected_game}" if detected_game else "No game detected" - } - except Exception as e: - raise HTTPException(status_code=500, detail=f"Error detecting game: {str(e)}") - -@app.get("/games/list") -def list_games(): - """List all available games.""" - try: - detection_games = get_detection_games() - csv_games = get_csv_games() - vector_games = list_available_games() - - return { - "status": "ok", - "detection_games": detection_games, - "csv_games": csv_games, - "vector_games": vector_games - } - except Exception as e: - raise HTTPException(status_code=500, detail=f"Error listing games: {str(e)}") - -# Knowledge Management endpoints -@app.post("/games/{game_name}/knowledge/process") -def process_game_knowledge(game_name: str): - """Process and vectorize knowledge for a specific game.""" - try: - # Validate CSV structure first - is_valid, errors = validate_csv_structure(game_name) - if not is_valid: - raise HTTPException(status_code=400, detail=f"Invalid CSV structure: {errors}") - - # Process and add to vector database - success = add_game_knowledge(game_name) - if success: - stats = get_game_stats(game_name) - return { - "status": "ok", - "message": f"Successfully processed knowledge for {game_name}", - "stats": stats - } - else: - raise HTTPException(status_code=500, detail="Failed to process game knowledge") - except HTTPException: - raise - except Exception as e: - raise HTTPException(status_code=500, detail=f"Error processing knowledge: {str(e)}") - -@app.post("/games/{game_name}/knowledge/search") -def search_game_knowledge(game_name: str, request: KnowledgeSearchRequest): - """Search knowledge base for a specific game.""" - try: - results = search_knowledge( - game_name=game_name, - query=request.query, - content_types=request.content_types, - limit=request.limit - ) - - return { - "status": "ok", - "game_name": game_name, - "query": request.query, - "results": results, - "total_results": len(results) - } - except Exception as e: - raise HTTPException(status_code=500, detail=f"Error searching knowledge: {str(e)}") - -@app.get("/games/{game_name}/knowledge/stats") -def get_game_knowledge_stats(game_name: str): - """Get statistics for a game's knowledge base.""" - try: - stats = get_game_stats(game_name) - return { - "status": "ok", - "game_name": game_name, - "stats": stats - } - except Exception as e: - raise HTTPException(status_code=500, detail=f"Error getting stats: {str(e)}") - -@app.get("/games/{game_name}/knowledge/validate") -def validate_game_csv(game_name: str): - """Validate CSV structure for a game.""" - try: - is_valid, errors = validate_csv_structure(game_name) - return { - "status": "ok", - "game_name": game_name, - "is_valid": is_valid, - "errors": errors - } - except Exception as e: - raise HTTPException(status_code=500, detail=f"Error validating CSV: {str(e)}") - -# Settings endpoints -@app.get("/settings/api-key") -def get_api_key_status(): - """Return whether an API key is configured (masked).""" - import os - key = os.getenv('GOOGLE_API_KEY') or "" - masked = (len(key) >= 8) - preview = f"{key[:4]}***{key[-4:]}" if masked else "" - return {"status": "ok", "configured": bool(key), "preview": preview} - -@app.post("/settings/api-key") -def update_api_key(req: ApiKeyRequest): - """Save API key to .env and reconfigure the chatbot model.""" - try: - key = (req.api_key or "").strip() - if not key: - raise HTTPException(status_code=400, detail="API key cannot be empty") - # Persist to .env - env_path = ".env" - # Load existing lines if any - lines = [] - if os.path.exists(env_path): - with open(env_path, 'r', encoding='utf-8') as f: - lines = f.read().splitlines() - # Update or append GOOGLE_API_KEY - found = False - for i, line in enumerate(lines): - if line.startswith("GOOGLE_API_KEY="): - lines[i] = f"GOOGLE_API_KEY={key}" - found = True - break - if not found: - lines.append(f"GOOGLE_API_KEY={key}") - with open(env_path, 'w', encoding='utf-8') as f: - f.write("\n".join(lines) + ("\n" if lines else "")) - # Reconfigure runtime - if not set_api_key(key): - raise HTTPException(status_code=500, detail="Failed to apply API key at runtime") - return {"status": "ok", "message": "API key updated"} - except HTTPException: - raise - except Exception as e: - raise HTTPException(status_code=500, detail=f"Error updating API key: {str(e)}") - -if __name__ == "__main__": - uvicorn.run(app, host="127.0.0.1", port=8000, reload=True) +app.include_router(chat.router, tags=["Chat"]) +app.include_router(screenshot.router, prefix="/screenshots", tags=["Screenshots"]) +app.include_router(game_detection.router, prefix="/games", tags=["Game Detection"]) +app.include_router(settings.router, prefix="/settings", tags=["Settings"]) \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index d6c1706..1cdf261 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,6 +25,7 @@ dependencies = [ "beautifulsoup4>=4.12.0", "pandas>=2.0.0", "lxml>=4.9.0", + "pytest>=8.4.2", ] [project.optional-dependencies] @@ -62,4 +63,4 @@ asyncio_mode = "auto" [tool.setuptools.packages.find] where = ["."] include = ["backend*"] -exclude = ["tests*", "vector_db*", "games_info*"] \ No newline at end of file +exclude = ["tests*", "vector_db*", "games_info*"] diff --git a/routers/__init__.py b/routers/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/routers/chat.py b/routers/chat.py new file mode 100644 index 0000000..713bc5e --- /dev/null +++ b/routers/chat.py @@ -0,0 +1,8 @@ +from fastapi import APIRouter +from services.chatbot import chat_with_gemini +from schemas.chat import ChatMessage +router = APIRouter() + +@router.post("/chat") +async def chat(message: ChatMessage): + return await chat_with_gemini(message.message, message.image_data) \ No newline at end of file diff --git a/routers/game_detection.py b/routers/game_detection.py new file mode 100644 index 0000000..27e2836 --- /dev/null +++ b/routers/game_detection.py @@ -0,0 +1,112 @@ +from services.game_detection import detect_current_game, get_available_games as get_detection_games +from services.knowledge_manager import get_available_games as get_csv_games, validate_csv_structure +from services.vector_service import add_game_knowledge, search_knowledge, get_game_stats, list_available_games +from schemas.game_detection import GameDetectionRequest +from schemas.knowledge_search import KnowledgeSearchRequest +from fastapi import APIRouter,HTTPException + +router = APIRouter() +# Game Detection endpoints +@router.post("/detect") +def detect_game(request: GameDetectionRequest): + """Detect current game from process/screenshot/message.""" + try: + detected_game = detect_current_game(request.message) + return { + "status": "ok", + "detected_game": detected_game, + "message": f"Detected game: {detected_game}" if detected_game else "No game detected" + } + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error detecting game: {str(e)}") + +@router.get("/list") +def list_games(): + """List all available games.""" + try: + detection_games = get_detection_games() + csv_games = get_csv_games() + vector_games = list_available_games() + + return { + "status": "ok", + "detection_games": detection_games, + "csv_games": csv_games, + "vector_games": vector_games + } + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error listing games: {str(e)}") + +# Knowledge Management endpoints +@router.post("/{game_name}/knowledge/process") +def process_game_knowledge(game_name: str): + """Process and vectorize knowledge for a specific game.""" + try: + # Validate CSV structure first + is_valid, errors = validate_csv_structure(game_name) + if not is_valid: + raise HTTPException(status_code=400, detail=f"Invalid CSV structure: {errors}") + + # Process and add to vector database + success = add_game_knowledge(game_name) + if success: + stats = get_game_stats(game_name) + return { + "status": "ok", + "message": f"Successfully processed knowledge for {game_name}", + "stats": stats + } + else: + raise HTTPException(status_code=500, detail="Failed to process game knowledge") + except HTTPException: + raise + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error processing knowledge: {str(e)}") + +@router.post("/{game_name}/knowledge/search") +def search_game_knowledge(game_name: str, request: KnowledgeSearchRequest): + """Search knowledge base for a specific game.""" + try: + results = search_knowledge( + game_name=game_name, + query=request.query, + content_types=request.content_types, + limit=request.limit + ) + + return { + "status": "ok", + "game_name": game_name, + "query": request.query, + "results": results, + "total_results": len(results) + } + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error searching knowledge: {str(e)}") + +@router.get("/{game_name}/knowledge/stats") +def get_game_knowledge_stats(game_name: str): + """Get statistics for a game's knowledge base.""" + try: + stats = get_game_stats(game_name) + return { + "status": "ok", + "game_name": game_name, + "stats": stats + } + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error getting stats: {str(e)}") + +@router.get("/{game_name}/knowledge/validate") +def validate_game_csv(game_name: str): + """Validate CSV structure for a game.""" + try: + is_valid, errors = validate_csv_structure(game_name) + return { + "status": "ok", + "game_name": game_name, + "is_valid": is_valid, + "errors": errors + } + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error validating CSV: {str(e)}") diff --git a/routers/screenshot.py b/routers/screenshot.py new file mode 100644 index 0000000..7176f55 --- /dev/null +++ b/routers/screenshot.py @@ -0,0 +1,51 @@ +from fastapi import APIRouter,HTTPException +from services.screenshot import start_screenshot_capture, stop_screenshot_capture, get_recent_screenshots, get_screenshot_by_id, get_screenshot_stats, delete_screenshot +router = APIRouter() +# Screenshot endpoints +@router.post("/start") +def start_screenshots(interval: int = 30): + """Start automatic screenshot capture.""" + start_screenshot_capture(interval) + return {"status": "ok", "message": f"Screenshot capture started with {interval}s interval"} + +@router.post("/stop") +def stop_screenshots(): + """Stop automatic screenshot capture.""" + stop_screenshot_capture() + return {"status": "ok", "message": "Screenshot capture stopped"} + +@router.get("/recent") +def get_recent_screenshots_endpoint(limit: int = 10, application: str = None): + """Get recent screenshots.""" + screenshots = get_recent_screenshots(limit=limit, application=application) + return {"status": "ok", "screenshots": screenshots} + +@router.get("/stats") +def get_screenshot_stats_endpoint(): + """Get screenshot statistics.""" + stats = get_screenshot_stats() + return {"status": "ok", "stats": stats} + +@router.get("/{screenshot_id}") +def get_screenshot_endpoint(screenshot_id: int): + """Get screenshot data by ID.""" + screenshot_data = get_screenshot_by_id(screenshot_id) + if screenshot_data: + import base64 + return {"status": "ok", "data": base64.b64encode(screenshot_data).decode('utf-8')} + else: + return {"status": "error", "message": "Screenshot not found"} + +@router.delete("/{screenshot_id}") +def delete_screenshot_endpoint(screenshot_id: int): + """Delete screenshot by ID.""" + try: + deleted = delete_screenshot(screenshot_id) + if deleted: + return {"status": "ok", "message": f"Deleted screenshot {screenshot_id}"} + else: + raise HTTPException(status_code=404, detail="Screenshot not found") + except HTTPException: + raise + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error deleting screenshot: {str(e)}") \ No newline at end of file diff --git a/routers/settings.py b/routers/settings.py new file mode 100644 index 0000000..7176c92 --- /dev/null +++ b/routers/settings.py @@ -0,0 +1,47 @@ +from fastapi import APIRouter, HTTPException +from services.chatbot import set_api_key +from schemas.settings import ApiKeyRequest +import os +router = APIRouter() +@router.get("/api-key") +def get_api_key_status(): + """Return whether an API key is configured (masked).""" + import os + key = os.getenv('GOOGLE_API_KEY') or "" + masked = (len(key) >= 8) + preview = f"{key[:4]}***{key[-4:]}" if masked else "" + return {"status": "ok", "configured": bool(key), "preview": preview} + +@router.post("/api-key") +def update_api_key(req: ApiKeyRequest): + """Save API key to .env and reconfigure the chatbot model.""" + try: + key = (req.api_key or "").strip() + if not key: + raise HTTPException(status_code=400, detail="API key cannot be empty") + # Persist to .env + env_path = ".env" + # Load existing lines if any + lines = [] + if os.path.exists(env_path): + with open(env_path, 'r', encoding='utf-8') as f: + lines = f.read().splitlines() + # Update or append GOOGLE_API_KEY + found = False + for i, line in enumerate(lines): + if line.startswith("GOOGLE_API_KEY="): + lines[i] = f"GOOGLE_API_KEY={key}" + found = True + break + if not found: + lines.append(f"GOOGLE_API_KEY={key}") + with open(env_path, 'w', encoding='utf-8') as f: + f.write("\n".join(lines) + ("\n" if lines else "")) + # Reconfigure runtime + if not set_api_key(key): + raise HTTPException(status_code=500, detail="Failed to apply API key at runtime") + return {"status": "ok", "message": "API key updated"} + except HTTPException: + raise + except Exception as e: + raise HTTPException(status_code=500, detail=f"Error updating API key: {str(e)}") \ No newline at end of file diff --git a/schemas/__init__.py b/schemas/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/schemas/chat.py b/schemas/chat.py new file mode 100644 index 0000000..3bf4a94 --- /dev/null +++ b/schemas/chat.py @@ -0,0 +1,6 @@ +from pydantic import BaseModel +from typing import Optional, List + +class ChatMessage(BaseModel): + message: str + image_data: Optional[str] = None \ No newline at end of file diff --git a/schemas/game_detection.py b/schemas/game_detection.py new file mode 100644 index 0000000..7427b54 --- /dev/null +++ b/schemas/game_detection.py @@ -0,0 +1,5 @@ +from pydantic import BaseModel +from typing import Optional, List + +class GameDetectionRequest(BaseModel): + message: Optional[str] = None \ No newline at end of file diff --git a/schemas/knowledge_search.py b/schemas/knowledge_search.py new file mode 100644 index 0000000..0cc3f6a --- /dev/null +++ b/schemas/knowledge_search.py @@ -0,0 +1,6 @@ +from pydantic import BaseModel +from typing import Optional, List +class KnowledgeSearchRequest(BaseModel): + query: str + content_types: Optional[List[str]] = None + limit: Optional[int] = 5 \ No newline at end of file diff --git a/schemas/settings.py b/schemas/settings.py new file mode 100644 index 0000000..b0f2706 --- /dev/null +++ b/schemas/settings.py @@ -0,0 +1,4 @@ +from pydantic import BaseModel +from typing import Optional, List +class ApiKeyRequest(BaseModel): + api_key: str \ No newline at end of file diff --git a/services/__init__.py b/services/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/backend/chatbot.py b/services/chatbot.py similarity index 95% rename from backend/chatbot.py rename to services/chatbot.py index 9082ddb..eec1f70 100644 --- a/backend/chatbot.py +++ b/services/chatbot.py @@ -2,11 +2,11 @@ import os import google.generativeai as genai from dotenv import load_dotenv -from .screenshot import get_recent_screenshots, get_screenshot_by_id, get_screenshot_stats -from .game_detection import detect_current_game -from .vector_service import search_knowledge +from services.screenshot import get_recent_screenshots, get_screenshot_by_id, get_screenshot_stats +from services.game_detection import detect_current_game +from services.vector_service import search_knowledge import base64 -import json + system_prompt_file = open("PROMPTS.txt","r") system_prompt = system_prompt_file.read() load_dotenv() diff --git a/backend/game_detection.py b/services/game_detection.py similarity index 100% rename from backend/game_detection.py rename to services/game_detection.py diff --git a/backend/knowledge_manager.py b/services/knowledge_manager.py similarity index 98% rename from backend/knowledge_manager.py rename to services/knowledge_manager.py index 0d40419..332d821 100644 --- a/backend/knowledge_manager.py +++ b/services/knowledge_manager.py @@ -239,8 +239,8 @@ def process_game_knowledge(self, game_name: str) -> Dict[str, List[Dict]]: time.sleep(1) # Be respectful to servers print(f"Processed {len(processed_knowledge['wiki'])} wiki entries, " - f"{len(processed_knowledge['youtube'])} YouTube entries, " - f"{len(processed_knowledge['forum'])} forum entries for {game_name}") + f"{len(processed_knowledge['youtube'])} YouTube entries, " + f"{len(processed_knowledge['forum'])} forum entries for {game_name}") return processed_knowledge diff --git a/backend/screenshot.py b/services/screenshot.py similarity index 100% rename from backend/screenshot.py rename to services/screenshot.py diff --git a/backend/vector_service.py b/services/vector_service.py similarity index 99% rename from backend/vector_service.py rename to services/vector_service.py index 7cbdd2c..289a8d5 100644 --- a/backend/vector_service.py +++ b/services/vector_service.py @@ -38,7 +38,7 @@ def _init_chroma_client(self): self.chroma_client = None def _init_embedding_model(self): - """Initialize Mistral client for embeddings using API key.""" + """Initialize the sentence transformer embedder""" try: # api_key = os.getenv('MISTRAL_API_KEY') self.embedding_model = SentenceTransformer("all-MiniLM-L6-v2") diff --git a/tests/unit/test_backend.py b/tests/unit/test_backend.py index 19c6eea..ca736b8 100644 --- a/tests/unit/test_backend.py +++ b/tests/unit/test_backend.py @@ -19,10 +19,10 @@ try: from backend.backend import app from backend.chatbot import chat_with_gemini, set_api_key - from backend.screenshot import get_recent_screenshots, get_screenshot_by_id, get_screenshot_stats, delete_screenshot - from backend.game_detection import detect_current_game, get_available_games as get_detection_games - from backend.knowledge_manager import get_available_games as get_csv_games, validate_csv_structure - from backend.vector_service import add_game_knowledge, search_knowledge, get_game_stats, list_available_games + from services.screenshot import get_recent_screenshots, get_screenshot_by_id, get_screenshot_stats, delete_screenshot + from services.game_detection import detect_current_game, get_available_games as get_detection_games + from services.knowledge_manager import get_available_games as get_csv_games, validate_csv_structure + from services.vector_service import add_game_knowledge, search_knowledge, get_game_stats, list_available_games except ImportError as e: pytest.skip(f"Backend modules not available: {e}", allow_module_level=True) diff --git a/tests/unit/test_game_detection.py b/tests/unit/test_game_detection.py index bdc6469..4391061 100644 --- a/tests/unit/test_game_detection.py +++ b/tests/unit/test_game_detection.py @@ -17,7 +17,7 @@ sys.path.insert(0, project_root) try: - from backend.game_detection import ( + from services.game_detection import ( GameDetection, detect_current_game, add_game_mapping, diff --git a/tests/unit/test_knowledge_manager.py b/tests/unit/test_knowledge_manager.py index 9fdad00..08e16a1 100644 --- a/tests/unit/test_knowledge_manager.py +++ b/tests/unit/test_knowledge_manager.py @@ -17,7 +17,7 @@ sys.path.insert(0, project_root) try: - from backend.knowledge_manager import ( + from services.knowledge_manager import ( KnowledgeManager, get_available_games, process_game_knowledge, diff --git a/tests/unit/test_screenshot.py b/tests/unit/test_screenshot.py index a03ff3d..2cead82 100644 --- a/tests/unit/test_screenshot.py +++ b/tests/unit/test_screenshot.py @@ -21,7 +21,7 @@ sys.path.insert(0, project_root) try: - from backend.screenshot import ( + from services.screenshot import ( ScreenshotCapture, start_screenshot_capture, stop_screenshot_capture, diff --git a/tests/unit/test_vector_service.py b/tests/unit/test_vector_service.py index 29bd5b2..3a05aad 100644 --- a/tests/unit/test_vector_service.py +++ b/tests/unit/test_vector_service.py @@ -18,7 +18,7 @@ sys.path.insert(0, project_root) try: - from backend.vector_service import ( + from services.vector_service import ( VectorService, add_game_knowledge, search_knowledge, diff --git a/uv.lock b/uv.lock index bd66483..66a2e7f 100644 --- a/uv.lock +++ b/uv.lock @@ -817,6 +817,7 @@ dependencies = [ { name = "pandas" }, { name = "pillow" }, { name = "psutil" }, + { name = "pytest" }, { name = "pywin32" }, { name = "requests" }, { name = "sentence-transformers" }, @@ -848,6 +849,7 @@ requires-dist = [ { name = "pandas", specifier = ">=2.0.0" }, { name = "pillow", specifier = ">=10.0.0" }, { name = "psutil", specifier = ">=5.9.0" }, + { name = "pytest", specifier = ">=8.4.2" }, { name = "pytest", marker = "extra == 'test'", specifier = ">=7.4.0" }, { name = "pytest-asyncio", marker = "extra == 'test'", specifier = ">=0.21.0" }, { name = "pytest-cov", marker = "extra == 'test'", specifier = ">=4.1.0" },