Finetuning prebuilt tree model issue

[Jelle-d]

Hi everyone,

I am new to coding and am trying to finetune the prebuilt tree model of DeepForest.
I am working in Google colab and here's my code: https://colab.research.google.com/drive/1GXKwKASSUb_xwW1ju6GN_X5tOzDVh55d?usp=sharing
With annotations:
result.json

And images like:

During hyperparameter tuning:, I get this output:

Validation Metrics: {'results': None, 'box_recall': 0, 'box_precision': nan, 'class_recall': None}
Fold F1-score: 0
GPU Memory Usage: 128MB / 15360MB (Free: 14966MB)
Starting Epoch 0...
GPU Memory Usage: 128MB / 15360MB (Free: 14966MB)
Epoch 0, Loss: None

Which is not supposed to happen, so the model probably is not finetuning at all (validation metrics, and loss are zero)?

However, the final output is still:

Best parameters: {'lr': 0.003993016688847723, 'batch_size': 12, 'epochs': 6}

But I don't know how trustworthy this is and where it comes from.
I do not understand what is going wrong, any help would be greatly appreciated!

[Jelle-d]

Hi everyone,
To make my script more manageable, I tried training without hyperparameter tuning. However, I encountered the following errors (see below). The most crucial one is 'No predictions made,' indicating that DeepForest (DF) is not detecting any bounding boxes in my validation dataset? This is puzzling because there are definitely enough tiles and visible trees in the dataset, and the DF model initially already predicted trees in the images.

UserWarning: Got processor for bboxes, but no transform to process it.

UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 3. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.

UserWarning: Converting predictions to GeoDataFrame using geometry column

FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

No predictions made

Does anyone know the problem (and solution)?

(here's my script: https://colab.research.google.com/drive/1kA-70HjYzYArdoyNWCnU1u9-ldAwL73v?usp=sharing)

[bw4sz]

Thanks for posting, let's take this nice and slow. Way before training, i'm worried about how the data was read in. Can you put the annotations file here, atleast one row of it? Let's make sure it can be read by deepforest.utilities.read_file. Then lets run the

from deepforest import main
from deepforest.visualize import plot_results
m = main.deepforest()
m.load_model("Weecology/deepforest-tree")
results = m.predict_tile(<path_to_sample_image>, patch_size=<we'll have to guess the ground resolution>)
plot_results(results)

As a service to the DeepForest open source community can you please suggest 2 ways in which the docs could have better helped you, so that the next user can learn faster.

1. What parts of the docs were clear to you?
2. What parts of the docs did you not understand?

Thank you.

[Jelle-d]

Hi, thank you for your answer!
When running your code on a random image of my annotations, this is the result:

So the model does predict objects (when I’m not finetuning it).
My annotations file was indeed a bit odd (label-studio had some weird export) so I changed it to have the “image_path, xmin, ymin, xmax, ymax, label” as a csv. As seen here:

(But still I get a loss of zero and ‘no predictions made’ )

[bw4sz]

• For the sake of ease, don't sample 30% of the data, just to make sure the model isn't crashing.
• Run 5 epochs with the larger dataset
• Out of pure courisity, not related, can you confirm that it trains fast with 11 workers and not 0.

Without the code in front of me i'll need to make a guess, but first asking what the evaluation score is, you ran so few epochs that it didn't trigger manually, looks like

DeepForest/src/deepforest/deepforest_config.yml

Line 60 in 1acab8b

val_accuracy_interval: 20

is 20, you can change that to lower, but also can call

m.trainer.validate()

to get a test evaluation score, that will help us understand what if any issue, or if that print statement means anything.

[Jelle-d]

Dear Ben,

I believe I’ve identified the source of the issue. It appears that several of the tree bounding box coordinates extended beyond the boundaries of their respective images. After removing those specific rows from the dataset, the model ran without errors. Below is the output of the successful run.

Regarding your earlier question: the documentation has been incredibly helpful—especially the Colab training tutorial, which was very clear and thorough. I was able to get quite far with very little prior experience, so thank you for making those resources available.

Million trees: As for my dataset: it consists of 3,409 tree bounding boxes on PNG images, each sized 640×640 pixels at an approximate resolution of 0.094 meters per pixel. While the tiles and bounding boxes are not georeferenced, the filenames do include the coordinates of the upper-left corner of each tile. If this format aligns with the goals of the MillionTrees dataset, I’d be happy to share.

Thanks again for your support and for building such a useful tool!

Training finished in 689.52 seconds
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃      Validate metric      ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│            iou            │    0.7691217064857483     │
│         iou/cl_0          │    0.7691217064857483     │
│            map            │    0.06474726647138596    │
│          map_50           │    0.11596134305000305    │
│          map_75           │    0.06223400682210922    │
│         map_large         │    0.08030504733324051    │
│        map_medium         │    0.06607016921043396    │
│       map_per_class       │           -1.0            │
│         map_small         │    0.04872728884220123    │
│           mar_1           │   0.052199412137269974    │
│          mar_10           │    0.46979472041130066    │
│          mar_100          │    0.5372434258460999     │
│     mar_100_per_class     │           -1.0            │
│         mar_large         │    0.6396825313568115     │
│        mar_medium         │    0.5405857563018799     │
│         mar_small         │    0.3512820601463318     │
│    val_bbox_regression    │    0.28236573934555054    │
│    val_classification     │    1.7882771492004395     │
└───────────────────────────┴───────────────────────────┘
[{'val_classification': 1.7882771492004395,
  'val_bbox_regression': 0.28236573934555054,
  'iou': 0.7691217064857483,
  'iou/cl_0': 0.7691217064857483,
  'map': 0.06474726647138596,
  'map_50': 0.11596134305000305,
  'map_75': 0.06223400682210922,
  'map_small': 0.04872728884220123,
  'map_medium': 0.06607016921043396,
  'map_large': 0.08030504733324051,
  'mar_1': 0.052199412137269974,
  'mar_10': 0.46979472041130066,
  'mar_100': 0.5372434258460999,
  'mar_small': 0.3512820601463318,
  'mar_medium': 0.5405857563018799,
  'mar_large': 0.6396825313568115,
  'map_per_class': -1.0,
  'mar_100_per_class': -1.0}]

[ethanwhite]

Glad to hear things are working for you @Jelle-d! I've opened an issue so that we throw a useful error in this case in the future - #1014

[bw4sz]

For sure, thanks for your efforts. We would be happy to include the dataset in MillionTrees, let's make an issue here -> and tell me how to access, cite it, whether it should be held back until the unpublished data version is ready, all are welcome to contribute to that manuscript, or if you are ready to share the data now for the public beta version.

https://github.com/weecology/MillionTrees/issues