Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
134 lines (94 loc) · 6.28 KB

CONTRIBUTING.md

File metadata and controls

134 lines (94 loc) · 6.28 KB

CONTRIBUTING

This document contains the list of issues, suggestions and improvements that can be added to this project and a proper guide on how to contribute to the project.

Registration

Important

All contributors must do the following to be eligible for MLSA X HACKTOBERFEST:

Guidelines

Adhere to Hacktober Fest Guidelines and maintain common etiquette of contributing to an open source project.

All contributors are requested to join this WhatsApp Group for all forms of communication regarding contributing to this repository.

List of major issues

Important

It is highly recommended for contributors to first have a look at the list of major issues work on them with higher priority.

Related to UI :-

  1. Make the window resizable and all window elements scalable.
  2. Add window control buttons for minimise and fullscreen (to be added after the previous implementation)
  3. Whenever you hover over the screenshot button, placeholder text is inserted in the chat box but it doesn't go away after you stop hovering, and you have to manually press backspace to remove the text.
  4. After adding your API Key from the overlay, once the overlay is closed and reopened later, it stops showing the placeholder text that your key has been added. Make the placeholder text persistent across sessions.

Frontend+Backend :-

  1. Chat history is not stored, if you try to follow up gemini with what you asked in the previous chat it will have 0 idea what you are talking about.

Tip

Here is a suggested solution: Store the recent chats of the user in a database, (we are already using ChromaDB, might as well use it), then when we give a new prompt to gemini, old chats are added to the prompt.

  • Only store the last 30 (or user specified) chats or so.
  • Every game will have its own database table, i.e. chats are stored on a per game basis.
  • Additional meta-data such as timestamp should also be stored so when user asked "What did I do yesterday?" it should be able to retrieve the screenshot from 24 hours ago etc.

Important

UI/UX Addition :

  • Add the ability to view the stored chats across various sessions/games from the overlay.
  • Add the configuration in the settings menu to read, delete and edit them.
  • Add a setting to set how many chats per game/session to store.

Backend :-

  1. Web Scrappers in knowledge_manager.py, extract_wiki_content() and extract_forum_content() sometimes gets blocked by certain websites, (namely the ones present in minecraft.csv)

Tip

Recommended Solutions :

  • Mask the scrappers to behave to more like human by introducing delays between different searches.
  • Use Proxies to circumvent IP bans.
  • Rotate a list of User Agents and headers.
  • Make it asynchronous using asyncio + httpx

List of Improvements and Suggestions

Tip

Feel free to give us any of your ideas, suggestions and feedback to add to this list.

  1. Youtube Video Indexing : Add the ability to index the description, audio transcriptions, titles and tags of youtube videos and recommend them to the user for a given prompt, from the list of videos in the CSV file.

Important

UI/UX Addition :

  • Add the ability to play the video within the overlay itself.
  • Add basic features play, pause and other functionalities.

Improvements to the UI

  1. Add Markdown support for Gemini's responses.
  2. Improve the chat Window UI such that it is easier to tell apart the messages of the user and Pixly.
  3. Add the ability to view images results from the web.

Long-term Goals

  1. Add the ability for users to add Custom Themes of the overlay, different themes belonging to different games, so when the overlay can automatically apply a certain theme when a particular game is detected.

Improvements to Backend :-

  1. Add more .csv entries about wikis, guides, youtube videos, forum posts about more games, especially single player story based titles like Elden Ring, Hollow Knight : Silksong, Black Myth: Wukong,Cyberpunk 2077
  2. Implement a better way to store screenshots:

Tip

Suggested Improvements :

  • Vectorise the screenshots as well, and store them in ChromaDB (make sure the content is encrypted as well)
  • Add tool calling for the agent to call a tool to retrieve the screenshot from a specific time or from a specific game.
  1. Add the ability for gemini to automatically perform web searches in case the query that the user is searching isn't present in the data base. If gemini finds said data then it adds to the vector db for future use.

Long term Goals

In game control

Add the ability for the agent to control the game and play the game for you and perform repetitive tasks, such as : Build a small house for me in minecraft. or Automatically plant and regrow my crops while I afk

Cross platform compatibility

Reliance on the win32 api for taking screenshots means we can't transition to a different platform, and are stuck with Windows for now. In the future we may wanna add cross platform compatibility with Linux.

Other Known Bugs and Issues : 🪲

Important

These are a bit obscure and their causes aren't known yet.

  1. Overlay hangs and then crashes when turning off the enable screenshots setting.
  2. game_detection.py, it reports the incorrect game being detected, in some cases.