Accumulate gradients on a single batch

[dvirginz]

Is there a way of propagating the same batch in consecutive steps using pl?
Given a network constructed of multiple unrelated objectives, running all objectives at once causes the GPU consumption to be really large, but, as the different networks and objectives are not related, I can pass each on in a different .backward() step, and aggregate to the optimizer once.
It resembles the accumulate gradients flag, but accumulation in pl is done on different batches.
Is there a way of doing so without canceling automatic backprop?

The desired pipe is:
For a given batch Bi:
A) global_step=i -> Run self.net1(B) and compute gradients
B) global_step=i+1 -> Run self.net2(B) and compute gradients
C) Accumulate and optimize

[ananthsub]

Is this intended to run on a single GPU or multiple GPUs?

There is manual_optimization which gives you utmost flexibility for how to process a single batch, as you control the forward, backward, step, and zero grad

[dvirginz]

Right, canceling the automatic optimization will sort this. I thought I can somehow use accumulate batches to pass the same batch twice