Skip to content

Commit 5a16289

Browse files
bw4szbw4sz
authored
Update report (#93)
* update release report * update leaderboard
1 parent 86d8b6a commit 5a16289

File tree

1 file changed

+51
-56
lines changed

1 file changed

+51
-56
lines changed

‎docs/leaderboard.md‎

Lines changed: 51 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@
22

33
# Tasks
44

5-
There are three tasks within the MillionTrees package.
6-
75
## Zero-shot
86

97
The first task is to create a zero-shot detection system to generalize across geography and aquisition conditions. Selected datasets are held out from training completely and used for evaluation in new conditions. This is a challenging task with no local training data.
@@ -16,7 +14,6 @@ The second task is to create the best global detector for individual trees given
1614

1715
Off the shelf tools often limit users for a single annotation type. We have 'point' models, 'box' models and 'polygon' models. To create truly global models for biological inference, we need models that can use all available data, not just one annotation geometry. In particular, polygon annotations are very time consuming to create, but are often desirable for downstream usecases. We opted against polygon training sources, for example polygons to points, as this is an unrealistic, or atleast, very uncommon downstream use case.
1816

19-
2017
### Boxes to Polygons
2118

2219
All box sources are used to train and predict all polygon sources. There is no local data from the test localities in train.
@@ -29,69 +26,27 @@ All point sources are used to train and predict all polygon sources
2926

3027
All point sources are used to train and predict all box sources.
3128

32-
33-
# Submissions
34-
35-
## Submit to the leaderboard
36-
37-
Once you have trained a model and evaluated its performance, you can submit your results to the MillionTrees leaderboard. Here's how:
38-
39-
1. Create a public repository with your code and model training scripts. Make sure to include:
40-
- Clear instructions for reproducing your results
41-
- Requirements file listing all dependencies
42-
- Training configuration files/parameters
43-
- Code for data preprocessing and augmentation
44-
- Model architecture definition
45-
- Evaluation code
46-
47-
2. Generate predictions on the test split:
48-
```python
49-
test_dataset = dataset.get_subset("test") # Use test split
50-
test_loader = get_eval_loader("standard", test_dataset, batch_size=16)
51-
52-
predictions = []
53-
for metadata, images, _ in test_loader:
54-
pred = model(images)
55-
predictions.append(pred)
56-
```
57-
58-
3. Save visual examples of your model's predictions:
59-
```python
60-
# Save a few example predictions
61-
dataset.visualize_predictions(
62-
predictions[:5],
63-
save_dir="prediction_examples/"
64-
)
65-
```
66-
67-
4. Submit a pull request to the [MillionTrees repository](https://github.com/weecology/MillionTrees) with:
68-
- Link to your code repository
69-
- Model description and approach
70-
- Performance metrics on test set
71-
- Example prediction visualizations
72-
- Instructions for reproducing results
73-
7429
# Results
7530

7631
## TreePoints
7732

7833
### Random
7934

80-
| Model | Task | Dataset | Avg Recall | Script |
35+
| Model | Task | Dataset | Counting MAE | Script |
8136
|---|---|---|---|---|
82-
| DeepForest baseline (`baseline_points.py`) | TreePoints | TreePoints | 0.000 | <small>`uv run python docs/examples/baseline_points.py --split-scheme random`</small> |
83-
| SAM3 (`sam3_points.py`) | TreePoints | TreePoints | 0.000 | <small>`uv run python docs/examples/sam3_points.py --device cuda --split-scheme random --hf-token $HF_TOKEN`</small> |
37+
| DeepForest baseline (`baseline_points.py`) | TreePoints | TreePoints | 26.183 | <small>`uv run python docs/examples/baseline_points.py --split-scheme random`</small> |
38+
| SAM3 (`sam3_points.py`) | TreePoints | TreePoints | 24.530 | <small>`uv run python docs/examples/sam3_points.py --device cuda --split-scheme random --hf-token $HF_TOKEN`</small> |
8439

8540
### Zero-shot
8641

87-
| Model | Task | Dataset | Avg Recall | Script |
42+
| Model | Task | Dataset | Counting MAE | Script |
8843
|---|---|---|---|---|
89-
| DeepForest baseline (`baseline_points.py`) | TreePoints | TreePoints | 0.000 | <small>`uv run python docs/examples/baseline_points.py --split-scheme zeroshot`</small> |
90-
| SAM3 (`sam3_points.py`) | TreePoints | TreePoints | 0.001 | <small>`uv run python docs/examples/sam3_points.py --device cuda --split-scheme zeroshot --hf-token $HF_TOKEN`</small> |
44+
| DeepForest baseline (`baseline_points.py`) | TreePoints | TreePoints | 50.744 | <small>`uv run python docs/examples/baseline_points.py --split-scheme zeroshot`</small> |
45+
| SAM3 (`sam3_points.py`) | TreePoints | TreePoints | 54.464 | <small>`uv run python docs/examples/sam3_points.py --device cuda --split-scheme zeroshot --hf-token $HF_TOKEN`</small> |
9146

9247
### Cross-geometry
9348

94-
| Model | Task | Dataset | Avg Recall | Script |
49+
| Model | Task | Dataset | Counting MAE | Script |
9550
|---|---|---|---|---|
9651
| DeepForest baseline (`baseline_points.py`) | TreePoints | TreePoints | 0.000 | <small>`uv run python docs/examples/baseline_points.py --split-scheme crossgeometry`</small> |
9752
| SAM3 (`sam3_points.py`) | TreePoints | TreePoints | 0.000 | <small>`uv run python docs/examples/sam3_points.py --device cuda --split-scheme crossgeometry --hf-token $HF_TOKEN`</small> |
@@ -102,17 +57,17 @@ Once you have trained a model and evaluated its performance, you can submit your
10257

10358
| Model | Task | Dataset | Avg Recall | Script |
10459
|---|---|---|---|---|
105-
| DeepForest baseline (`baseline_boxes.py`) | TreeBoxes | TreeBoxes | | <small>`uv run python docs/examples/baseline_boxes.py --split-scheme random`</small> |
106-
| SAM3 (`sam3_boxes.py`) | TreeBoxes | TreeBoxes | | <small>`uv run python docs/examples/sam3_boxes.py --device cuda --split-scheme random --hf-token $HF_TOKEN`</small> |
60+
| DeepForest baseline (`baseline_boxes.py`) | TreeBoxes | TreeBoxes | 0.397 | <small>`uv run python docs/examples/baseline_boxes.py --split-scheme random`</small> |
61+
| SAM3 (`sam3_boxes.py`) | TreeBoxes | TreeBoxes | 0.165 | <small>`uv run python docs/examples/sam3_boxes.py --device cuda --split-scheme random --hf-token $HF_TOKEN`</small> |
10762
| Torchvision Faster R-CNN (`torchvision_fasterrcnn_treeboxes.py`) | TreeBoxes | TreeBoxes | | <small>`uv run python docs/examples/torchvision_fasterrcnn_treeboxes.py --split_scheme random`</small> |
10863
| Ultralytics YOLO (`yolo_treeboxes.py`) | TreeBoxes | TreeBoxes | | <small>`uv run python docs/examples/yolo_treeboxes.py --split_scheme random --weights yolov8n.pt`</small> |
10964

11065
### Zero-shot
11166

11267
| Model | Task | Dataset | Avg Recall | Script |
11368
|---|---|---|---|---|
114-
| DeepForest baseline (`baseline_boxes.py`) | TreeBoxes | TreeBoxes | 0.546 | <small>`uv run python docs/examples/baseline_boxes.py --split-scheme zeroshot`</small> |
115-
| SAM3 (`sam3_boxes.py`) | TreeBoxes | TreeBoxes | 0.212 | <small>`uv run python docs/examples/sam3_boxes.py --device cuda --split-scheme zeroshot --hf-token $HF_TOKEN`</small> |
69+
| DeepForest baseline (`baseline_boxes.py`) | TreeBoxes | TreeBoxes | 0.534 | <small>`uv run python docs/examples/baseline_boxes.py --split-scheme zeroshot`</small> |
70+
| SAM3 (`sam3_boxes.py`) | TreeBoxes | TreeBoxes | 0.208 | <small>`uv run python docs/examples/sam3_boxes.py --device cuda --split-scheme zeroshot --hf-token $HF_TOKEN`</small> |
11671
| Torchvision Faster R-CNN (`torchvision_fasterrcnn_treeboxes.py`) | TreeBoxes | TreeBoxes | | <small>`uv run python docs/examples/torchvision_fasterrcnn_treeboxes.py --split_scheme zeroshot`</small> |
11772
| Ultralytics YOLO (`yolo_treeboxes.py`) | TreeBoxes | TreeBoxes | 0.004 | <small>`uv run python docs/examples/yolo_treeboxes.py --split_scheme zeroshot --weights yolov8n.pt`</small> |
11873

@@ -148,3 +103,43 @@ Once you have trained a model and evaluated its performance, you can submit your
148103
| DeepForest baseline (`baseline_polygons.py`) | TreePolygons | TreePolygons | | <small>`uv run python docs/examples/baseline_polygons.py --split-scheme crossgeometry`</small> |
149104
| SAM3 (`sam3_polygons.py`) | TreePolygons | TreePolygons | | <small>`uv run python docs/examples/sam3_polygons.py --device cuda --split-scheme crossgeometry --hf-token $HF_TOKEN`</small> |
150105

106+
# Submissions
107+
108+
## Submit to the leaderboard
109+
110+
Once you have trained a model and evaluated its performance, you can submit your results to the MillionTrees leaderboard. Here's how:
111+
112+
1. Create a public repository with your code and model training scripts. Make sure to include:
113+
- Clear instructions for reproducing your results
114+
- Requirements file listing all dependencies
115+
- Training configuration files/parameters
116+
- Code for data preprocessing and augmentation
117+
- Model architecture definition
118+
- Evaluation code
119+
120+
2. Generate predictions on the test split:
121+
```python
122+
test_dataset = dataset.get_subset("test") # Use test split
123+
test_loader = get_eval_loader("standard", test_dataset, batch_size=16)
124+
125+
predictions = []
126+
for metadata, images, _ in test_loader:
127+
pred = model(images)
128+
predictions.append(pred)
129+
```
130+
131+
3. Save visual examples of your model's predictions:
132+
```python
133+
# Save a few example predictions
134+
dataset.visualize_predictions(
135+
predictions[:5],
136+
save_dir="prediction_examples/"
137+
)
138+
```
139+
140+
4. Submit a pull request to the [MillionTrees repository](https://github.com/weecology/MillionTrees) with:
141+
- Link to your code repository
142+
- Model description and approach
143+
- Performance metrics on test set
144+
- Example prediction visualizations
145+
- Instructions for reproducing results

0 commit comments

Comments
 (0)