Initial commit of our automated illusion generator package. Much to be done.

Author: aribenjamin

autogen/README.md

@@ -0,0 +1,34 @@
+# Automated generation of novel illusions
+
+This package leverages a new theory of visual illusions to generate new illusions of new types.
+
+In human vision, there is a robust connection between perceptial bias (i.e. an illusion) and discrimination errors (i.e. uncertainty).
+(see [this paper](https://www.sas.upenn.edu/~astocker/lab/publications-files/journals/PNAS2017/Wei_Stocker2017.pdf)or
+[this paper too](https://www.sas.upenn.edu/~astocker/lab/publications-files/journals/NN2015/Wei_Stocker2015b.pdf)).
+
+
+Since pretrained deep neural networks show similar uncertainty about the world (in a Fisher information sense), 
+we can use them as proxies for human uncertainty. This allows us to find new illusions in an automated fashion.
+
+
+#### Algorithm
+To create an image with an illusory percept for thing x (e.g. curvature of some line, lightness, etc.)
+
+1) Define a parameterized image generator G with x as a smoothly-varying input parameter. 
+2) For any generated image, estimate to Fisher of a pretrained DNN w/r/t x
+3) Sample from the density of G parameters that yield images for which the Fisher for x changes quickly, using deep density techniques (e.g. GANs)
+
+
+#### Organization
+
+In `image_generators.py`, we define our image generators. Each subclass of `BaseImageGenerator` has methods to generate
+an image deterministically and smoothly from "context" inputs and a 1D "theta" input.
+
+In `fisher_estimation.py`, we define scripts that calculates the Fisher information of a pretrained deep neural network
+layer as a function of "theta", given some "context".
+
+In `density_estimation.py`, we define scripts that can sample from the density of "context" variables to find values 
+where the Fisher information w/r/t theta has a large value. 
+
+
+

autogen/density_estimation.py

@@ -0,0 +1,26 @@
+import torch
+
+""" These scripts aim to find values of 'context' that have highly sloped Fisher 
+information in pretrained network representations. 
+
+There are a few possibilities. All of them would benefit from speed, and smoothness of the map from context to Fisher.
+In order of complexity:
+
+1.  Optimization
+----------------
+For low-dimensional contexts, we could use an out-of-the-box, general purpose optimizer, e.g. in scipy. This approach
+is more promising when the context is not expected to have image-like structure (see 'Deep Image Prior'). 
+
+Initially, we'll simply optimize a neural network to transform random noise into the context parameters, with the loss
+being the squared slope of the Fisher at that value of context. 
+If degenerate modes are found, we might have to impose an entropy constraint on the output. 
+
+2. Sampling
+-----------
+Using the derivative of the Fisher as the likelihood, or energy, we can also apply MCMC sampling methods. This is 
+probably more likely to work in higher-dimensional situations.
+
+If this shows slow convergence, we can attempt new MCMC methods that additionally incorporate a model that selects
+which things are sampled. (e.g. https://papers.nips.cc/paper/7099-a-nice-mc-adversarial-training-for-mcmc.pdf)
+
+"""

autogen/fisher_estimation.py

@@ -0,0 +1,97 @@
+import numpy as np
+import torch
+
+class BaseDifferentiableImageGenerator(object):
+
+    """Base class for all image generators that are written in pytorch and differentiable into theta.
+    All classes should output an image of ImageNet dimensions (224x224).
+    """
+
+    def __init__(self):
+
+        pass
+
+    def generate_image(self, theta, context):
+        """To be redefined in any subclass.
+
+        Deterministically generates an image as a function of theta and context.
+
+        Context and theta should both smoothly relate to the image. Small changes in
+        theta and changes in context should cause small changes in the output image.
+        For two values of theta but the same context, the two output images should be
+        the same in all manners except for theta.
+
+        Methods need to be written in pytorch, such that they take a Variable and produce an
+        image that is differentiable with respect to theta
+
+        """
+        raise NotImplementedError()
+
+
+class BaseNonDifferentiableImageGenerator(object):
+    """Base class for all image generators that aren't differentiable.
+
+    All classes should output an image of ImageNet dimensions (224x224).
+
+    When possible, all methods should still use torch methods and not numpy or scipy methods, for speed."""
+
+    def __init__(self):
+        pass
+
+    def generate_image(self, theta, context):
+        """To be redefined in any subclass.
+
+        Deterministically generates an image as a function of theta and context.
+
+        Context and theta should both smoothly relate to the image. Small changes in
+        theta and changes in context should cause small changes in the output image.
+        For two values of theta but the same context, the two output images should be
+        the same in all manners except for theta.
+
+        """
+        raise NotImplementedError()
+
+class OneCurvedLineGenerator(BaseDifferentiableImageGenerator):
+    """This is designed to create illusions like the Herring illusion.
+    """
+
+    def __init__(self, n_lines = 5):
+        super(OneCurvedLineGenerator, self).__init__()
+        self.n_lines = n_lines
+
+
+    def generate_image(self, theta, context):
+        """
+    Theta is the curvature of the central horizontal line.
+    Context are parameters that describe the overlaid lines.
+    e.g. a list of midpoints and orientations, for a total of n_lines x 3 parameters. """
+
+        raise NotImplementedError()
+
+
+
+class CentralPixelGenerator(BaseDifferentiableImageGenerator):
+    """This is designed to generate images with a central block of pixels whose lightness is an illusions.
+    """
+
+    def __init__(self, n_pixels_blocks_per_side = 5):
+        """
+
+
+        :param n_pixels_blocks_per_side: an odd integer
+        """
+        super(CentralPixelGenerator, self).__init__()
+        assert n_pixels_blocks_per_side % 2 == 1, "n_pixels_blocks_per_side must be odd"
+
+        self.n_pixels_blocks_per_side = n_pixels_blocks_per_side
+
+
+    def generate_image(self, theta, context):
+        """
+        :param theta: the lightness of the central pixel block.
+        :param context: the lightness of the surrounding pixels.
+        :return: A 224x224x3 image
+        """
+        assert len(context) == self.n_pixels_blocks_per_side**2-1
+
+        raise NotImplementedError()

autogen/image_generators.py

@@ -0,0 +1,3 @@
+import argparse
+
+"""Runs it all"""

autogen/main.py

@@ -0,0 +1,12 @@
+import numpy as np
+
+
+"""Unit tests for any image generator.
+
+Includes: 
+ - tests of differentiability w/r/t theta
+ - output sizes
+ 
+
+
+"""