Adding DeepForest multi agent with HuggingFace models implementation

Author: Samia35-2973

.gitignore

@@ -16,3 +16,16 @@ eggs/
 .installed.cfg
 *.egg
 MANIFEST
+
+
+# Environment
+.env
+
+
+# Testing
+.pytest_cache/
+
+
+# Project specific
+lightning_logs/
+.gradio/

README.md

@@ -1 +1,77 @@
-# deepforest-agent
+# DeepForest Multi-Agent System
+
+The DeepForest Multi-Agent System provides ecological image analysis by orchestrating multiple AI agents that work together to understand ecological images. Simply upload an image of a forest, wildlife habitat, or ecological scene, and ask questions in natural language.
+
+## Installation
+
+### 1. Clone the repository
+
+```bash
+git clone https://github.com/weecology/deepforest-agent.git
+cd deepforest-agent
+```
+
+### 2. Create and activate a Conda environment
+
+```bash
+conda create -n deepforest_agent python=3.12.11
+conda activate deepforest_agent
+```
+
+### 3. Install dependencies
+
+```bash
+pip install -r requirements.txt
+pip install -e .
+```
+
+### 4. Configure the HuggingFace Token
+Create a `.env` file in the root directory of the deepforest-agent project and add your HuggingFace token like below:
+
+```bash
+HF_TOKEN="your_huggingface_token_here"
+```
+
+You can obtain your token from [HuggingFace Access Token](https://huggingface.co/settings/tokens). Make sure the Token type is "Write".
+
+## Usage
+
+The DeepForest Agent runs through a Gradio web interface. To start the interface, execute:
+
+```bash
+python -m deepforest_agent.app
+```
+
+A link like http://127.0.0.1:7860 will appear in the terminal. Open it in your browser to interact with the agent. A public Gradio link may also be provided if available.
+
+
+## Features
+
+- **Multi-Species Detection**: Automatically detects trees, birds, and livestock using specialized DeepForest models
+- **Tree Health Assessment**: Identifies alive and dead trees using DeepForest Tree Detector whenever user asks.
+- **Visual Analysis**: Dual analysis of original and annotated images using Qwen2.5-VL-3B-Instruct model
+- **Memory Context**: Maintains conversation history for contextual understanding across multiple queries
+- **Ecological Insights**: Synthesizes detection data with visual analysis for comprehensive ecological understanding
+- **Streaming Responses**: Real-time updates as each agent processes your query
+
+
+## Requirements
+
+### Hardware Requirements
+- **GPU**: GPU with at least 24GB VRAM (recommended for optimal performance)
+- **Storage**: At least 16GB free space for model downloads
+
+### API Requirements
+- **HuggingFace Token**: Required for model access. 
+
+
+## Models Used
+
+- **SmolLM3-3B**: For Memory, Detector, and Ecology agents for text understanding and generation
+- **Qwen2.5-VL-3B-Instruct**: Used in Visual agent for multimodal image-text understanding
+- **DeepForest Models**: For tree, bird, and livestock detection. Also used for alive/dead tree classification.
+
+
+## Multi-Agent Workflow
+
+[![](https://mermaid.ink/img/pako:eNq9Vd1y2jgUfhWNbkuykJhu8HTaoRiStCUECElTkwvVPmBtbcmVZBJKMrPPso-2T7LHsg0m6XZmb5ZhPPb5-c53_qQNDWQI1KWLWN4HEVOGXHlzQfDX9WcaFJmlsWShJucJW8IdOTh4S977U9CaS0GGTKBUkZ4CZkCTUn5XILy31r0CZwoCUcYZqHWp7lm1549UEIE2ihmJZgY5aHIj1becUmlaPD3r0PeHkEi1Jt0lCPPmq_rtbVeweP0D4wdSrEBpZnJyEdeIuG5Yk1TBistMkxXXGYuJAp1KoUE3iJEyJgGLY11G69s4g80Z08TjCgJDukLfg7JICyUTclZgv3uq8xvkfo-3_ekjOfUHYIKI9BjmFhIPDKLkpDxmWBnm1IY58_uBjOWyns8EAuAr0C4pU8W8DDwY8sqqS9BwC4rJZLGp6J9Z3HN_arArCRlwrA6ZlPnevSR8MXokH_zroi62lJrrGhvEyQKTKQw5ykyamRLjg43z0Z8AKoV2rfHff_5VjAr5jnDcrMkCu-oBpAOJNM3WaCci8usfeY0FS0Bv9SMT4dQUKk3uuYmwClKFmI2pmZW0WUm7pPbRUvu0KZiMCybW51TK8Bml_SZ-2tZk-N87U7EJQ543Bl93TQqx88_M9knX49shuvBrRSpGCInvyOBeIRG7Xd_zvSKvfsnqXypeRr-wJRv5l0queAi1dl5V-4FzxrQUXCy3um6aKpkqjj15uUcjC3np9yIIvhUwvRzGzm9pc2ltxhsrw7V61o3xrhqTctCqnbJ4k73JH297N_X7DxBkyOpFCXfn09TGvsJFQX0FRbjYI3hljWZV8J9ELZ4Ta3f9f81MKa1lt-9VcpsVrOpEh1Z088vj4bqwqbvdWNFnf5aGebN7ETPkHPNQCxZUbp-tza3vcZ3GDGsghDRoHRZHQlWwW2v2pYLaPxzJECcMG3JHG3SpeEjdBYs1NGgCKmH5N93kMHOKB0QCc-riq4AM7494TufiCf1SJr5ImVAXTy70VDJbRtVHZoN6nC0VS7bgCm8nUD2ZCUPd1tHvR0cWhbob-kDdo-bxYdtpHZ80j09OnHaz87pB19Q9aHVOnEOn5TTb7Vb79XHH6Tw16A8b2TlsNpsOyh38O81Op0EhzPMaFvetvXaf_gGHt3cm?type=png)](https://mermaid.live/edit#pako:eNq9Vd1y2jgUfhWNbkuykJhu8HTaoRiStCUECElTkwvVPmBtbcmVZBJKMrPPso-2T7LHsg0m6XZmb5ZhPPb5-c53_qQNDWQI1KWLWN4HEVOGXHlzQfDX9WcaFJmlsWShJucJW8IdOTh4S977U9CaS0GGTKBUkZ4CZkCTUn5XILy31r0CZwoCUcYZqHWp7lm1549UEIE2ihmJZgY5aHIj1becUmlaPD3r0PeHkEi1Jt0lCPPmq_rtbVeweP0D4wdSrEBpZnJyEdeIuG5Yk1TBistMkxXXGYuJAp1KoUE3iJEyJgGLY11G69s4g80Z08TjCgJDukLfg7JICyUTclZgv3uq8xvkfo-3_ekjOfUHYIKI9BjmFhIPDKLkpDxmWBnm1IY58_uBjOWyns8EAuAr0C4pU8W8DDwY8sqqS9BwC4rJZLGp6J9Z3HN_arArCRlwrA6ZlPnevSR8MXokH_zroi62lJrrGhvEyQKTKQw5ykyamRLjg43z0Z8AKoV2rfHff_5VjAr5jnDcrMkCu-oBpAOJNM3WaCci8usfeY0FS0Bv9SMT4dQUKk3uuYmwClKFmI2pmZW0WUm7pPbRUvu0KZiMCybW51TK8Bml_SZ-2tZk-N87U7EJQ543Bl93TQqx88_M9knX49shuvBrRSpGCInvyOBeIRG7Xd_zvSKvfsnqXypeRr-wJRv5l0queAi1dl5V-4FzxrQUXCy3um6aKpkqjj15uUcjC3np9yIIvhUwvRzGzm9pc2ltxhsrw7V61o3xrhqTctCqnbJ4k73JH297N_X7DxBkyOpFCXfn09TGvsJFQX0FRbjYI3hljWZV8J9ELZ4Ta3f9f81MKa1lt-9VcpsVrOpEh1Z088vj4bqwqbvdWNFnf5aGebN7ETPkHPNQCxZUbp-tza3vcZ3GDGsghDRoHRZHQlWwW2v2pYLaPxzJECcMG3JHG3SpeEjdBYs1NGgCKmH5N93kMHOKB0QCc-riq4AM7494TufiCf1SJr5ImVAXTy70VDJbRtVHZoN6nC0VS7bgCm8nUD2ZCUPd1tHvR0cWhbob-kDdo-bxYdtpHZ80j09OnHaz87pB19Q9aHVOnEOn5TTb7Vb79XHH6Tw16A8b2TlsNpsOyh38O81Op0EhzPMaFvetvXaf_gGHt3cm)

data/AWPE Pigeon Lake 2020 DJI_0005.JPG

@@ -0,0 +1,64 @@
+[project]
+name = "deepforest_agent"
+version = "0.1.0"
+description = "AI Agent for DeepForest object detection"
+authors = [
+    {name = "Your Name", email = "you@example.com"}
+]
+requires-python = ">=3.12"
+readme = "README.md"
+dependencies = [
+    "accelerate",
+    "albumentations<2.0",
+    "deepforest",
+    "fastapi",
+    "geopandas",
+    "google-genai",
+    "google-generativeai",
+    "gradio",
+    "gradio-image-annotation",
+    "langchain",
+    "langchain-community",
+    "langchain-google-genai",
+    "langchain-huggingface",
+    "langgraph",
+    "matplotlib",
+    "numpy",
+    "num2words",
+    "openai",
+    "opencv-python",
+    "outlines",
+    "pandas",
+    "pillow",
+    "plotly",
+    "pydantic",
+    "pydantic-settings",
+    "pytest",
+    "pytest-cov",
+    "python-dotenv",
+    "pyyaml",
+    "qwen-vl-utils",
+    "rasterio",
+    "requests",
+    "scikit-image",
+    "seaborn",
+    "shapely",
+    "streamlit",
+    "torch",
+    "torchvision",
+    "tqdm",
+    "transformers",
+    "bitsandbytes",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pre-commit",
+    "pytest",
+    "pytest-profiling",
+    "yapf"
+]
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"

data/OSBS_029.tif

@@ -0,0 +1,41 @@
+accelerate
+albumentations<2.0
+deepforest
+fastapi
+geopandas
+google-genai
+google-generativeai
+gradio
+gradio-image-annotation
+langchain
+langchain-community
+langchain-google-genai
+langchain-huggingface
+langgraph
+matplotlib
+numpy
+num2words
+openai
+opencv-python
+outlines
+pandas
+pillow
+plotly
+pydantic
+pydantic-settings
+pytest
+pytest-cov
+python-dotenv
+pyyaml
+qwen-vl-utils
+rasterio
+requests
+scikit-image
+seaborn
+shapely
+streamlit
+torch
+torchvision
+tqdm
+transformers
+bitsandbytes