DAG container

[ikostrikov]

Add a direct acyclic graph container so we could builld a network as in Caffe and Lasagne.

Example:

model = bb8.DAG()

fc1 = model.add('input', bb8.Linear(32, 64))
relu1 = model.add(fc1, bb8.ReLU())

[lucasb-eyer]

Agreed, and with the current design it should not even be hard to do.

And then we or someone else can build higher level functions to read cuda-convnet/caffe model files and instantiate such a model, but who'd want such a thing, hehe.

[ikostrikov]

We need to discuss how to implement this DAG container. I suggest to keep all symbolic variables hidden and simply work with some unique ids that is generated by the "add" function.

Also I think that we need to handle multiple inputs and multiple outputs for this container.

[lucasb-eyer]

General

Agreed on hiding all symbolic variables. But I think instead of using automatic unique IDs, we should let the user give unique names to all layers:

model = bb8.DAG()
model.add('input', bb8.Linear(32, 64), 'fc1')
model.add('fc1', bb8.Softmax(), 'sm')

or, with keyword arguments:

model.add(in='fc1', fn=bb8.Softmax(), out='sm')

Multiple inputs

A nice example of using this would be a depth-from-stereo algorithm which needs two pictures as input. With the name giving, this should be relatively easy, something like this would be nice:

model = bb8.DAG()
model.add_input('left image')
model.add_input('right image')
model.add('left image', bb8.Linear(32*32, 64), 'fcL1')
model.add('right image', bb8.Linear(32*32, 64), 'fcR1')
model.add(['fcL1', 'fcL2'], bb8.Mux(), 'mux')
model.add('mux', bb8.Linear(64+64, 10), 'fc2')
model.add('fc2', bb8.Softmax(), 'sm')

Multiple Outputs

A use-case for this (vision again) would be human attributes, e.g. for a picture say whether the person is male/female and also has glasses or not.

For this, we would also need a kind of weighted-sum-of-costs cost, which I think both of us had in our toolboxes. Then, it could simply be that the user calls model.add_output('fc2', bb8.Softmax(), 'sm') instead of model.add(...).

Thoughts?

[lucasb-eyer]

Addendum: and if we make two additional things:

1. Make add return the output-name, and
2. Make the out parameter of add default to None and generate an automatic unique ID if it is None.

Then it is also possible to use it exactly as in your first example. If you agree and want, I can write up a proposal implementation when I get time (maybe this evening).

[ikostrikov]

I agree with your suggestions and with addendum as well.