|
1 | | -# deepforest-agent |
| 1 | +# DeepForest Multi-Agent System |
| 2 | + |
| 3 | +The DeepForest Multi-Agent System provides ecological image analysis by orchestrating multiple AI agents that work together to understand ecological images. Simply upload an image of a forest, wildlife habitat, or ecological scene, and ask questions in natural language. |
| 4 | + |
| 5 | +## Installation |
| 6 | + |
| 7 | +### 1. Clone the repository |
| 8 | + |
| 9 | +```bash |
| 10 | +git clone https://github.com/weecology/deepforest-agent.git |
| 11 | +cd deepforest-agent |
| 12 | +``` |
| 13 | + |
| 14 | +### 2. Create and activate a Conda environment |
| 15 | + |
| 16 | +```bash |
| 17 | +conda create -n deepforest_agent python=3.12.11 |
| 18 | +conda activate deepforest_agent |
| 19 | +``` |
| 20 | + |
| 21 | +### 3. Install dependencies |
| 22 | + |
| 23 | +```bash |
| 24 | +pip install -r requirements.txt |
| 25 | +pip install -e . |
| 26 | +``` |
| 27 | + |
| 28 | +### 4. Configure the HuggingFace Token |
| 29 | +Create a `.env` file in the root directory of the deepforest-agent project and add your HuggingFace token like below: |
| 30 | + |
| 31 | +```bash |
| 32 | +HF_TOKEN="your_huggingface_token_here" |
| 33 | +``` |
| 34 | + |
| 35 | +You can obtain your token from [HuggingFace Access Token](https://huggingface.co/settings/tokens). Make sure the Token type is "Write". |
| 36 | + |
| 37 | +## Usage |
| 38 | + |
| 39 | +The DeepForest Agent runs through a Gradio web interface. To start the interface, execute: |
| 40 | + |
| 41 | +```bash |
| 42 | +python app.py |
| 43 | +``` |
| 44 | + |
| 45 | +A link like http://127.0.0.1:7860 will appear in the terminal. Open it in your browser to interact with the agent. A public Gradio link may also be provided if available. |
| 46 | + |
| 47 | +**Sample Recording of Running the System:** [Drive Link](https://drive.google.com/file/d/1gNMn-xJd48Ld3TZU4oiYvTbiWaiLsc8G/view?usp=sharing) |
| 48 | + |
| 49 | + |
| 50 | +### How to Use |
| 51 | + |
| 52 | +1. Upload an ecological image (aerial/drone photography works best) |
| 53 | +2. Ask questions about wildlife, forest health, or ecological patterns. For example: |
| 54 | + - How many trees are detected, and how many of them are alive vs dead? |
| 55 | + - How many birds are around each dead tree? |
| 56 | + - What objects are in the northwest region of the image? |
| 57 | + - Do any birds overlap with livestock in this image? |
| 58 | + - What percentage of the image is covered by trees vs birds vs livestock? |
| 59 | +3. Get comprehensive analysis combining computer vision and ecological insights. The gallery shows the annotated image with objects and the detection monitor presents the summary of DeepForest detection. |
| 60 | + |
| 61 | + |
| 62 | +## Features |
| 63 | + |
| 64 | +- **Multi-Species Detection**: Automatically detects trees, birds, and livestock using specialized DeepForest models |
| 65 | +- **Tree Health Assessment**: Identifies alive and dead trees using DeepForest Tree Detector whenever user asks. |
| 66 | +- **Visual Analysis**: Dual analysis of original and annotated images using Qwen2.5-VL-3B-Instruct model |
| 67 | +- **Memory Context**: Maintains conversation history for contextual understanding across multiple queries |
| 68 | +- **Tiling Image for Visual Agent:** Larger images are tiled and processed individually for the visual agent. |
| 69 | +- **R-Tree Spatial Indexing:** Stores DeepForest Results in an R-Tree spatial index structure and use spatial queries to retrieve relevant information and present it to the user. |
| 70 | +- **Ecological Insights**: Synthesizes detection data with visual analysis and memory context for comprehensive ecological understanding |
| 71 | +- **Streaming Responses**: Real-time updates as each agent processes your query |
| 72 | + |
| 73 | + |
| 74 | +## Requirements |
| 75 | + |
| 76 | +### Hardware Requirements |
| 77 | +- **GPU**: GPU with at least 24GB VRAM (recommended for optimal performance). The system is optimized for GPU execution. Running on CPU will take significantly longer processing times |
| 78 | +- **Storage**: At least 35GB free space for model downloads |
| 79 | + |
| 80 | +### API Requirements |
| 81 | +- **HuggingFace Token**: Required for model access. |
| 82 | + |
| 83 | + |
| 84 | +## Image Processing Times |
| 85 | + |
| 86 | +- **Standard Images**: Most ecological images process within 30 seconds on GPU |
| 87 | +- **Large GeoTIFF Files**: Larger geospatial images may require significant time for complete analysis |
| 88 | + |
| 89 | + |
| 90 | +## Models Used |
| 91 | + |
| 92 | +- **SmolLM3-3B**: For Memory Agent to get context, and for Detector Agent to call the tool with appropriate parameters |
| 93 | +- **Qwen2.5-VL-3B-Instruct**: Used in Visual agent for multimodal image-text understanding |
| 94 | +- **Llama-3.2-3B-Instruct**: For Ecology agents for text understanding and generation |
| 95 | +- **DeepForest Models**: For tree, bird, and livestock detection. Also used for alive/dead tree classification. |
| 96 | + |
| 97 | + |
| 98 | +## Multi-Agent Workflow |
| 99 | + |
| 100 | +[](https://mermaid.live/edit#pako:eNqlV9ty4jgQ_RWVt2pexmQI9_hhtwiQK4SLgVyceVBMO7giLEq2kzAk_75tSTYiM9k8LA8UpruPuk93H9tby-cLsBzrUdD1kky79xHBT9ubxSDIbM04XcTkfEUf4Scplf4mx15HAE2AuBDHIY_IN3IehUlIWfgL_0zQ9FNhHEv_jkJyIUKccQpio80dae56Q-EvIU4ETbhwyCBlSVhqP0KUkGsungLGXzJUkegw9d2VwT1vACsuNkT6O8RdcdYfVEvVY-3ck24nW-12RmPSDQX4CWlH8QuIf95N0JPM--0W4jdy6vVeMSV0nHLOSIdijuS8q2FPJeyZN4FEhPAMyr4gXUgQOyPligosKHzOuTiTEedez-eMPxYJ9xldUVI9qGDKeWbuJkqQkDDWoecy9M47CSPKyATiNY9iIC9hsiS6rg6PEnjdZ0gVc8XfyIU3D-MUY9sIsEHg_PTxC0SleX9H14U86nJ7kjKmer6LcVPfx44HKdsn7XJHWt9zw-iRwcfQDl-tGRRzcakzIyUyHA5ITwgu3sjAO6GMPVD_ySHTkEHpmMZIaQ6iYwcyw6t8BtVBmXusCBnRxF8SF0dRB1zJgKEncXBAe9gpGYATuabYI2D5QA6l68jDdB_CCPNHEqQnco5Tmacwkm59k4O-_GuM8xBzlsoB6CzBfyIBFzgUsD7hAkccOQwT-hCyMMnPHMvIyVYVMsYuoY2cco6VX3WJAahiGeyzPym67HruU7g2TyumUZ_lyrOmXp8_Euk7ARrjLGnzVJpnelhKw4htdi1EXpc_fz9Ytn3u_XYolv3ZSs7lMdfeKUSQ0Z8vGJItO6iSwjnS_tfS_2a7c1PLts_DTZ4ODpVa1rMweSO3v63ofi9vdqOoogZhjBW127j-4KeY3X_wqZWyrWTx2Jukkek-QF1lsUMSAfDjIRSLHwz1IE64_5QLpFbIzo6MnYK46WpFcbe_4fpwscDlTyBPu6O1s-u5SHUxoCSMDLnSvl0llbdmzrdq0kfepJRlR9w1zQQchXwBr0g97kaSrsn3Pwka0blyke-DWojxOF8c5w9VfC_OmACjmSlehuu8nrFeg8mOiEwzBCwhirMzPxdW9T2T8a7rjUS21R_D9_VxMtHeei30Xky9fTXFnLVu7v74Ko-pXqLZTug_aO6e4rvIPl3tZn2m6pjPvfwm8EvJkM4g63DCyf6dIN8rvVjXnkLd3Smm_Aki8rBRP_K10nt1o0domoqoKDSTravsh2bEvG3rxblRU3XrzdYL82lAPgDIoY2eQcSy1biLOPYFwq0av7s7rxvGa0Y3xfx-R7oiniEslLTHxuAMUBV2U6e-7-4UlPlfnGxQs9skCBlz_oLDoB6AaelqS1CFelA3Lb3cEqCtbFoucrRWUIeWaZl_GoN7oU0-1MA3TdhnjVcOKsGhabrLgw6DFhyZdfmMxnEXArKSTVGPSObhNj5EYYezy6NWOb8svYSLZOlU1q8fYJ7lcJswqroCpubToP4lzEIL_v_OJ1Z9LhbGJK-AgqNDaFS_ggK1fHu1SaYLnHL5qNFqfYXDjUfTvakpcI78SvPhs9IMNNKzT83GmaYL-9Lu2wP7yh7aI7MtptPcvrbbbfv42O50bNT0PdpNx9HIHo9t1LgPfJo-s5k9R8DrPaJMh67tuvZ0at_c2LitJg2WjW8K4cJyAspisK0ViBXNrq1tBnBvoWyt4N5y8GcEKQaxe-s-ese4NY3uOF9ZTiJSjBQ8fVzmF6lUkW5I8TVkVYALfGkA0eFplFhOrXwoMSxna71azmGzfFArN8qNZr1Rb7RqRw3b2lhOqVKttA6alWat1qgf1hvVZu3dtn7JcxsH5VqzUa81Kq1mvdpoVGwLFpmmDNQrkHwTev8XjGAuiw) |
0 commit comments