Integrating Elevation Data (DEM) into Tree Detection — Is it possible or recommended?

[maraosoc]

Hi DeepForest community,

I'm currently using DeepForest in a project based on identifying and counting citrus trees (orange and lemon) using high-resolution RGB aerial imagery, covering approx—93 hectares of mixed agricultural terrain.
I would like to improve the model's inference by incorporating elevation information —Canopy Height Models (CHMs). So my questions are:

• Has anyone in the community integrated elevation data into DeepForest pipelines before?
• Are there recommended practices or workarounds for including this kind of information?
• Would it be possible to fine-tune the model using stacked RGB + elevation (4-band) inputs somehow, even if (as I've seen) the model only receives 3-band data?

Any suggestions or shared experiences would be greatly appreciated!

Note: Visit the repo of my project for further info / data samples

[bw4sz]

We have attempted the 4 band strategy a number of times, and I think even some students wrote a paper about it, we have no evidence it outperforms just fine-tuning the model on a few annotations. The challenge is 1) you lose the value of most of the model weights, since that massive pretraining doesn't know anything about canopy height data, 2) how to normalize and concat the layers. Personally, I found this to be hard, and @MarconiS and I tried many norm strategies, quite a few years ago. Never found anything that seemed convincing.

Now, that does NOT mean it won't work, or can't work, or there is anything wrong with the idea. We sunk a bunch of time because we too thought it was a good idea. Only that we have no evidence that the fusion wasn't worth just annotating new data for RGB only. In general there must be some value to sensor fusion, including CHM or point clouds, but we haven't seen it beyond a couple small examples. I'll look for the conference paper if I can find it from a few years ago.