How can we interpret model hidden states?

[amcarter]

At our last meeting (2/15) we discussed comparing hidden state 3 to the LAI of the different sites.

Here, four focal sites for which LAI was available, we observed that H3 was similar for the first three, but quite different for the coastal plain site 01466500 (lower right)

The LAI for these four sites does not explain this difference.

The coastal plain site (upper left in this figure) does have the highest summer LAI of the four, but it is highly seasonal, not explaining the constant H3 state above

[galengorski]

This is an interesting paper that is under review at HESS: Hydrological concept formation inside long short-term memory (LSTM) networks.

Motivating question: Is the model learning hydrologic processes that are not explicitly included in the training dataset?

• The authors train a single LSTM on 669 catchments to predict streamflow from meteorological variables and watershed attributes
• They then extract the cell states from each timestep
• Then run a linear regression on cell state time series and independent hydrologic observational time series of soil moisture, and snow depth, they call this linear regression a "probe"
  • Basically, how well can a linear combination of cell states recreate an independent time series of a hydrologically relevant process

• The key findings are:
  • they can represent basin specific soil moisture reasonably well, but shallow much better than deeper
  • in catchments with snow they can represent snow depth, but they had to first identify which catchments these were
  • the findings were robust to model initialization with different random seeds, and different "probe" architecture

• A couple of nuggets that I thought were interesting:
  • By using a linear probe to relate cell states to the predicted variable (soil moisture or snow depth) they can interpret the weights of the linear probe to see which cell states are most important
  • Nice discussion of cell states: "There is no reason why the LSTM should store that information in a single cell, rather than distribute information across multiple cells. In fact, the LSTM could also model a process as the difference of two (or more cells) or any other combination of multiple different cells. Indeed it is likely that information is distributed due to the process of dropout. Using dropout randomly sets certain weights to zero during training in order to prevent the neural network overfitting, preventing the co-adaptation of weights such that one weight “corrects” the influence (Srivastava et al., 2014). However, this also means that the network can potentially learn the same process in two different places in the network, since the network must be robust to those weights being “switched off” when dropped out"
  • They raise the question of using the probe approach in ungauged basins, if an LSTM can generalize to an unseen basin, maybe the probe weights could as well

This is getting long so I'll end there, but I thought this might have some relevance to how we are thinking about cell states