Extend Autoupdates to Batch inference

[skoghoern]

For the epistemic prior implementations (given fixed matrices - agent without parameter learning) it seems to me that being able to access the marginals on the variables from the previous VMP iteration is enough to recompute the updated epistemic priors.
further it seems to be that this could be done relatively easy by adapting the batch.jl inference execution by accepting autoupdates similar as streaming.jl and fetching/pushing the latest marginals/messages exactly once at the start of each VMP iteration loop.
what do you guys think? @wouterwln @bvdmitri

[wouterwln]

I think this isn't true; I think to recompute the epistemic priors, you need access to the region beliefs (node local joints) which is something else than the independent marginals, and also different from the product between these marginals and the transition/observation tensor.

[skoghoern]

hmm okay. could you briefly explain why this is? (i think i must have missed something then)

[skoghoern]

ah, could it be that I mixed up the prior state as fixed in 1-step planning vs. variable in future rollout over several steps?

1-Step Planning (Fixed x_{t-1}):
$$E_{q(x_t \mid x_{t-1}, u_t)} \left[ -\log q(x_t \mid x_{t-1}, u_t) \right]$$

Future Rollout (Variable/Joint Prior):
$$E_{q(x_t, x_{t-1} \mid u_t)} \left[ -\log q(x_t \mid x_{t-1}, u_t) \right]$$

[wouterwln]

The reason we can't simply "fetch the latest marginals" from a previous iteration to recompute epistemic priors is that these priors are functions of the region beliefs (local joints) of a specific factor.

In a factor graph, the region belief $q_{f}(x, y)$ for a factor $f(x, y)$ is calculated as the product of the factor and the incoming messages $\overleftarrow{\mu}$ from the connected edges:

$$q_{f}(x, y) \propto f(x, y) \overleftarrow{\mu}(x) \overleftarrow{\mu}(y)$$

If you try to reconstruct this using the converged marginals $q(x)$ and $q(y)$ from the previous VMP iteration, you are effectively calculating:

$$q_{approx}(x, y) \propto f(x, y) q(x) q(y) \Bigg(= f(x,y) \big(\overleftarrow{\mu}(x) \overrightarrow{\mu}(x)\big) \big(\overleftarrow{\mu}(y)\overrightarrow{\mu}(y)\big) = q_{f}(x,y) \overrightarrow{\mu}(x) \overrightarrow{\mu}(y)\Bigg)$$

Here, I've taken notational liberty to let $\overleftarrow{\mu}$ indicate incoming messages, and $\overrightarrow{\mu}$ for outbound messages. This is a technical error. Because $q(x)$ already contains information from $f(x, y)$ (via the outgoing message $\overrightarrow{\mu}(x)$ ), multiplying them again "double-counts" the factor's influence. To get the true local joint, you need the "messages-in," which represent the neighbors' beliefs before they incorporate the information from $f$.

[skoghoern]

thank you for the intuitive clarification @wouterwln.
so if i understood correctly it seems that my error was to resolve $E_{q(x_t, x_{t-1} \mid u_t)} \left[ -\log q(x_t \mid x_{t-1}, u_t) \right]$ to
$$\sum_{x_{t-1} \in \mathcal{X}} \color{red}{q(x_{t-1})} \color{black}{H_q(x_t \mid x_{t-1}, u_t)}$$ instead of $$\sum_{x_{t-1} \in \mathcal{X}} \color{green}{q(x_{t-1} \mid u_t)} \color{black}{H_q(x_t \mid x_{t-1}, u_t)}$$ where $q(x_{t-1} \mid u_t)$ could be seen as the incoming message $\overleftarrow{\mu}$

so could my assumption be correct only for the special case, when we use mean-field factorization $q(x_{t-1}, u_t) = q(x_{t-1}),q(u_t)$ where $q(x_{t-1} \mid u_t) = q(x_{t-1})$ and (intuitively speaking) where we use the marginal $q(x_{t-1})$ instead of incoming message $\overleftarrow{\mu}$ to compute messages?

[skoghoern]

i guess as you stated initially, even the mean-field is not a special case for using the product of marginals and the transition/observation tensor, since the tensors e.g. $p(x_t \mid x_{t-1}, u_t)$ only represent the forward pass and we would omit the important information being passed backwards from e.g. the preference prior at $x_T$, which instead should be included in the belief $q(x_t \mid x_{t-1}, u_t)$

[bvdmitri]

That is a bit outside of the scope of @autoupdates to change the priors between VMP iterations. It is not about batch.jl or streaming.jl, since streaming.jl does not allow to update the priors in between iterations as well. It does only update in between receiving new observations, but for a single observation you can still do multiple VMP iterations and @autoupdates will not be called. Since batch.jl operates on a single batch (which you consider just a single big big big observation) hence there is no need to call autoupdate or support it there.

It also changes the optimization procedure in such a way that it becomes difficult (or impossible?) to prove that it will even converge, hence I would consider doing that "unsafe". Though if you want to experiment, you can still do that manually with custom callbacks that have access to the entire model, you can manually query all the region beliefs and update the priors manually (I think @wouterwln does something similar in his paper, but with meta structures)

[skoghoern]

@_convergence issue: in fact, as i understand this is the known limitation about the approach (which wouter also points out in their paper). the problem is that we "need" to define the priors using variational beliefs, and wouters idea was to solve this by initializating and updating them through VMP iterations.

thanks for the hint about the callbacks, i will dig into it.

[bvdmitri]

true, I'm working on the reimplementation of the callbacks system to support #566 . Should be ready in a week or two to play around. With the new system it should in principle be possible to "overwrite" messages on the fly way easier than in the current implementation

[skoghoern]

awesome. looking forward :D