WIP: Add mimir logger

[dmitriy-serdyuk]

First working draft of a mimir logger.

• Figure out if (un)picking works. Currently I just save the kwargs during picking and recreate logger during unpicking.
• Save 1d numpy arrays in a more readable format.
• Add some tests.
• Add documentation.

@rizar , @bartvm

[bartvm]

kwargs should be self.all_kwargs I guess?

[dmitriy-serdyuk]

Oops, it should be state.

[dmitriy-serdyuk]

@bartvm , seems the log writes only the last two lines to the file with an option max_len=2. I supposed, that this argument concerns only the information stored in memory.

[bartvm]

That doesn't sound right, max_len should only affect the entries kept in memory. Did you try it without your wrapper?

[dmitriy-serdyuk]

I fixed that problem, it was my bug.

I checked locally pickling/unpickling and it works fine, but Travis doesn't like it...

[bartvm]

I think the problem is that the log doesn't get closed. Until the log gets closed, gzlog maintains a lock file (log.jsonl.lock), and if you try to open a second log with the same filename it will wait for up to 5 minutes to obtain the lock.

If you don't call close explicitly the lock will only be released when close is called by io.RawIOBase's __del__ method (when it gets garbage collected), but if multiple tests are run in a single Python interpreter, I guess it's guesswork when this actually happens.

[dmitriy-serdyuk]

My implementation doesn't allow to access log entries more than 3 iterations before. It seems for me as a reasonable solution for many tasks with a large log.

But it means, that I cannot run all the tests (some of them are trying to check that the whole log is fine). I can see two solutions:

• Disable these tests for Mírmir log.
• Allow the access loading the whole log into memory as a separate Logger object.

Neither seems nice to me, what do you prefer, guys?

[bartvm]

An alternative solution would be to make maxlen a configuration setting with a default of 3. For those tests you can then temporarily change it from 3 to None so that the entire log gets kept?

[dmitriy-serdyuk]

Seems, that no:

In [4]: a = numpy.array(1)

In [5]: isinstance(a, numpy.generic)
Out[5]: False

[bartvm]

Right, that's not a scalar but a 0D array. I can add that case (if a.ndim == 0) to Mimir I guess.

Just wondering, where in Blocks are 0D arrays created? Reduction operations like sum and mean should generally return scalars instead of 0D arrays.

[bartvm]

Also, in general I'm not sure whether the numpy.asscalar call is a good idea. The problem is that this works on any array with a.size == 1. So if you have log entries with 1D arrays of variable length (e.g. np.array([1]) and np.array([2, 3])) some will be turned into Python scalars and others won't be, which could be messy when deserializing. By checking if ndim == 0 you'd avoid this.

[dmitriy-serdyuk]

As I can see, theano always returns 0d arrays even if the function is compiled with tensor.scalar type.

[bartvm]

Looks like it. Seems like an inconsistency between Theano and NumPy:

In [1]: from theano import tensor

In [2]: import numpy as np

In [3]: x = tensor.vector('x')

In [4]: x.mean().eval({x: np.random.rand(10)})
Out[4]: array(0.40431822667528144)

In [5]: np.random.rand(10).mean()
Out[5]: 0.43414875634541017

[bartvm]

bartvm/mimir@cd8167f

[rizar]

I am not sure that it is worth to run all these additional tests every time. We can at least skip Python 2.7 tests for Blocks and only do them for Blocks-examples.

[dmitriy-serdyuk]

I agree, that's too much. I'll remove some tests after I'll be sure that everything works.

[dmitriy-serdyuk]

I restarted the master branch build and it also fails, Travis has some problem again, since there's no g++ somehow.

[bartvm]

You don' have to do this, Python will complain if a user passes filename as a positional as well as a keyword argument.

I think it would be cleaner to just use __init__(self, filename='log.jsonl.gz', maxlen=21, formatter=None, **kwargs) though and then call PicklableLogger(filename=filename, maxlen=maxlen, formater=formatter, **kwargs). Better for default arguments to be in the signature than hidden in the code.

[bartvm]

@dmitriy-serdyuk Please rebase on master and give Travis another try!

[bartvm]

@dmitriy-serdyuk Did this ever end up working?

[dmitriy-serdyuk]

Not yet =)

I promise that I'll find time to finish it. I'm starting to think that it's much easier to change the logger interface than trying to mimic the default dict.