Recurrent Dqn

[lake4790k]

One central element of the Atari DQN is the use of 4 consecutive frames as input making the state more Markov, ie. having the vital dynamic movement information. This paper http://arxiv.org/abs/1507.06527v3 discusses DRQN: the multiframe input can be substituted with LSTM with the same effect (but no systematic advantage for one or the other). Also the Deepmind async paper mentions using LSTM instead of multi frame inputs for more challenging visual domains (Torcs and Labyrinth).

I think this would fit well in this codebase, I'll try to contribute this at one point.

[Kaixhin]

Yep a switch for using a DRQN architecture would be great. For now I'd go for using histLen as the number of frames to use BPTT on for a single-frame DRQN. Would be good to base it on the rnn library, especially since it now has the optimised SeqLSTM.

[lake4790k]

This is the Caffe implementation from the paper:
https://github.com/mhauskn/dqn/tree/recurrent

Altough Caffe I never looked at probably will help.

[lake4790k]

@Kaixhin I see you started working on this, cool. I'll have some time now, so I'll look at the multigpu and async modes.

[Kaixhin]

@lake4790k Almost have something working. Disabling this line lets the DRQN train, as otherwise it crashes here, somehow propagating a batch of size 20 forward but expecting the normal batch size of 32 backwards.

I'm new to the rnn library, so let me know if you have any ideas. Performance is considerably slower, which will be due to having to process several time steps sequentially. This is in line with Appendix B in that paper though.

[lake4790k]

@Kaixhin Awesome! I have no experience with rnn either, I will need to study it to have an idea. I have two 980TIs and will be able to run longer experiments to see if it goes anywhere.

[Kaixhin]

@lake4790k I'd have to delve into the original paper/code, but it looks like they train the network every step (as opposed to every 4). This seems like it'll be a problem for BPTT. In any case if you haven't used rnn before I'll focus on this.

[lake4790k]

@Kaixhin cool, I'll have my hands full with async for now, but in the meantime I'll be able to help with running longer rdqn experiments on my workstation when you think it's worth trying.

[Kaixhin]

Here's the result of running ./run.sh demo -recurrent true, so I'm reasonably confident that the DRQN is capable of learning, but I'm not testing this further for now so I'm leaving this issue open. In any case, I still haven't solved this issue (which I mentioned above).

[Kaixhin]

Pinging @JoostvDoorn since he's contributed to rnn and may have ideas about the minibatch problem/performance improvements/whether it's possible to save and restore state before and after training (and if that should be done since the parameters have changed slightly).

[JoostvDoorn]

@Kaixhin I will have a look later.

[lake4790k]

@Kaixhin I'm not getting the error you mentioned when doing validation on the last batch with size 20 when running demo. I'm using the master code which has sequencer:remember('both') enabled. You mention you had to disable that to not crash...? master runs fine for me as it is.

[JoostvDoorn]

I think this is in the rnn branch. This may or may not be a bug when using FastLSTM with the nngraph version. Setting nn.FastLSTM.usenngraph = false changed the error for me, but I only got the chance to look at this for a moment.

[lake4790k]

ok so there are two issues:

1. nn.FastLSTM.usenngraph = true
nngraph/gmodule.lua:335: split(4) cannot split 32 outputs
   this is issue in both rnn and master
2. nn.FastLSTM.usenngraph = false
Wrong size for view. Input size: 20x1x3. Output size: 32x3
   this is only in rnn, because @Kaixhin fixed Agent.valMemory and validate() #16 in master (but not in rnn) that returns before doing the backward during validation, because it is not even needed, so maybe no issue after all?

[Kaixhin]

1. With nn.FastLSTM.usenngraph = true, I get the same error as @lake4790k. This seems to be incompatibility with nngraph's gmodule in evaluate mode Element-Research/rnn#172. Which is a shame, as apparently it's significantly faster with this flag enabled (see Slow LSTM speed relative to theano. Element-Research/rnn#182).
2. Yes, so if you remove the return on line 374 in master then it fails. So I consider this a bug, albeit one that is being hidden by that return - why is this occurring even when states is 20x4x1x24x24 and QCurr is 20x1x3? If the error is dependent on previous batches then the learning must be incorrect. I was wrong and removing sequencer:remember('both') doesn't stop the crash.

[lake4790k]

@Kaixhin re: 2. agree, this error is bad, so returning before is not a solution. I'm not sure if learning is bad with the normal batch sizes, could be only not handling a batch size change somewhere properly. I tried an isolated FastLSTM+Sequencer net, there switching batch sizes worked fine, weird. I'm looking adding LSTM to async, once I get that working will experiment with this further.