Merge pull request #41 from pakozm/devel

Release v0.3.0-beta, from Devel branch

Author: pakozm

CHANGELIST.md

@@ -1,82 +1,123 @@
 ChangeList
 ==========
 
-Master branch unstable release
-------------------------------
+Master branch release
+---------------------
 
-- Added `matrix.abs`.
-- Solved bug at method `matrix::best_span_iterator::setAtWindow`. Becaose of It
-  the method didn't works when the matrix was a sub-matrix (slice) of other
-  matrix.
-- Added `matrix.join` method.
-- Added PCA-GS algorithm for efficient computation of PCA (iterative algorithm),
+v0.3.0-beta relase
+------------------
+
+### API Changes
+
+- Added `loss` parameter to `trainable.supervised_trainer` methods.
+- Added `optimizer` parameter to `trainable.supervised_trainer` methods.
+- Added `ann.optimizer` package, which has the implementation of weight update
+  based on weight gradient. So, the ANN components only compute gradients.
+  This allows to implement different optimization methods (as "Conjugate
+  Gradient", or "Linear Search Brack-Propagation") with the same gradient
+  computation.
+- Loss functions `ann.loss` API has been changed, now the loss computation is
+  done in two separated steps:
+      - `matrix = loss:compute_loss(output,target)`: which returns a matrix with
+        the loss of every pair of patterns, allowing to perform several loss
+		computations without taken them into account.
+      - `matrix = loss:accum_loss(matrix)`: which accumulates the loss in the
+	    internal state of the class.
+- Added a new version of `loss` function, which computes mean and
+  sample variance of the loss. Besides, the loss computation is done
+  with doubles, being more accurated than before.
+- `replacement` parameter in SDAE doesn't force `on_the_fly` parameter, they are
+  independent.
+- SDAE training has been changed in order to allow the use of LUA datasets,
+- Replaced `cpp_class_binding_extension` by `class_extension` function,
+  adding Lua classes support besides to CPP binded classes.
+- Modified `class` and `class_instance` functions to be more homogeneous
+  with C++ binding.
+- Added support for GZipped matrices load and save from C++, so functions
+  `matrix.savefile` and `matrix.loadfile` (and its correspondence for complex
+  numbers, double, int32, and char) were removed. Methods `matrix.fromFilename`
+  and `matrix.toFilename` accept '.gz' extension.
+
+### Packages rename
+
+- Changed package `sdae` to `ann.autoencoders`.
+- Changed package `loss_functions` to `ann.loss`.
+- Splitted `mathcore` package into `mathcore` and `complex` packages.
+- Renamed `math` package to `mathcore` to avoid the collision with Lua standard
+  math library.
+
+### New features
+
+- April-ANN is deployed as a standalone executable and as a shared library for
+  Lua 5.2.
+- Modified `lua.h` to incorporate the GIT commit number in the disclaimer.
+- Added Lua autocompletion when readline is available.
+- Implemented SignalHandler class in C++.
+- Added `signal.register` and `signal.receive` functions to Lua.
+- Added to `matrix` the methods `map`, `contiguous`, `join`, `abs`, `tan`,
+  `atan`, `atanh`, `sinh`, `asin`, `asinh`, `cosh`, `acos`, `acosh`, `fromMMap`,
+  `toMMap`, `div`, `max`, `min`.
+- Added `iterator` class, which is a wrapper around Lua iterators, but
+  provides a more natural interface with functional programming procedures
+  as `map`, `filter`, `apply`, or `reduce`.
+- Added methods `iterate`, `field`, `select` to iterator Lua class.
+- `table.insert` returns the table, which is useful for reduction operations.
+- Added `table` method to `iterator` class.
+- Added naive `L1_norm` regularization.
+- Added `dataset.clamp`.
+- Added `mathcore.set_mmap_allocation` function, which allows to forces the
+  use of mmap for `matrix` memory allocation.
+- Added `ann.components.slice`.
+- Added GS-PCA algorithm for efficient computation of PCA (iterative algorithm),
   `stats.iterative_pca` Lua function.
-- Added `fromMMap` and `toMMap` for `matrix` class, currently only with floats.
 - Added basic MapReduce implementation in Lua.
 - Added `stats.correlation.pearson` Lua class.
 - Added `stats.bootstrap_resampling` function.
-- Added method `iterate` to iterator Lua class.
-- Modified `lua.h` to incorporate the GIT commit number in the disclaimer.
-- `table.insert` returns the table, which is useful for reduction operations.
-- Added `table` method to `iterator` class.
-- Added a new version of `loss` function, which computes mean and
-  sample variance of the loss. Besides, the loss computation is done
-  with doubles, being more accurated than before.
-- Added `loss`  parameter to `trainable.supervised_trainer` methods.
 - Added `math.add`, `math.sub`, `math.mul`, `math.div` functions.
-- Methods `field` and `select` added to `iterator` class.
-- Added `div` method to `matrix`.
-- Added `signal.register` and `signal.receive` functions to Lua.
-- Implemented SignalHandler class in C++.
 - `trainable` and `ann.mlp.all_all` are using `matrix:to_lua_string()`
   method.
 - Added method `to_lua_string()` in all matrix types, so the method produce
   a Lua chunk which is loadable and produce a matrix.
-- Added `iterator` class, which is a wrapper around Lua iterators, but
-  provides a more natural interface with functional programming procedures
-  as `map`, `filter`, `apply`, or `reduce`.
 - Added serialization to `parallel_foreach`, allowing to produce outputs which
   could be loaded by the caller process.
 - Declaration of `luatype` function as global, it wasn't.
-- Added BIND_STRING_CONSTANT to luabind, so it is possible to export C string
-  constants to Lua.
-- Removed warning of clang about unused variables, adding a new macro
-  `UNUSED_VARIABLE(x)` defined in the header `utils/c_src/unused_variable.h`.
 - Added `iterable_map` and `multiple_ipairs` functions to the Lua utilities.
-- `replacement` parameter in SDAE doesn't force `on_the_fly` parameter, they are
-  independent.
-- SDAE training has been changed in order to allow the use of LUA datasets,
-  improving-
-- Solved bugs at Matrix template constructor which affects to `rewrap` lua
-  method, and to select method, which affects to `select` lua method.
-- Configured Lua package path to be in /usr/ instead of /usr/local/. It is
-  the default place in Ubuntu.
-- Replaced `cpp_class_binding_extension` by `class_extension` function,
-  adding Lua classes support besides to CPP binded classes.
-- Modified `class` and `class_instance` functions to be more homogeneous
-  with C++ binding.
 - Added SubAndDivNormalizationDataSet, applies a substraction and a division of
   the feature vectors.
 - Added stepDataset.
+
+### Bugs removed
+
+- Solved bug at `luabind_template.cc`, which introduced spurious segmentation
+  faults due to Lua execution of garbage collection in the middle of a
+  `lua_pushClassName`.
 - Solved bug at glob function.
+- Solved bug at matrix iterators operator=.
+- Solved bug at method `matrix::best_span_iterator::setAtWindow`. Becaose of It
+  the method didn't works when the matrix was a sub-matrix (slice) of other
+  matrix.
+- Solved bugs at Matrix template constructor which affects to `rewrap` lua
+  method, and to select method, which affects to `select` lua method.
 - Added binarizer::init() to a binded static_constructor, it is needed to
   execute init() before decode/encode double numbers, because of endianism.
 - Solved bug at constString when extracting double numbers in binary format.
-- Added max and min methods over a given dimension for `matrix`.
 - MacOS compilation problems solved.
+- Solved problems with CUDA, it is working again.
+- Dynamic loading of C modules is working now.
+
+### C/C++ code changes
+
+- Added BIND_STRING_CONSTANT to luabind, so it is possible to export C string
+  constants to Lua.
+- Removed warning of clang about unused variables, adding a new macro
+  `UNUSED_VARIABLE(x)` defined in the header `utils/c_src/unused_variable.h`.
 - Matrix fromString and toString Lua methods have been improved to write/read
   directly from Lua string buffer, so the memory print has been reduced.
 - The C++ routines to write and read files is generalized to work with streams,
   under the BufferedStream template, and it is instantiated to FILE and gzFile
   formats.
 - Added sanity check to cross-entropy and multi-class cross-entropy loss
   functions, to detect the use of non logarithmic outputs.
-- Solved problems with CUDA, it is working again.
-- Dynamic loading of C modules is working now.
-- Added support for GZipped matrices load and save from C++, so functions
-  `matrix.savefile` and `matrix.loadfile` (and its correspondence for complex
-  numbers, double, int32, and char) were removed. Methods `matrix.fromFilename`
-  and `matrix.toFilename` accept '.gz' extension.
 
 v0.2.1-beta relase
 ------------------

DISCLAIMER.lua

@@ -0,0 +1,4 @@
+return {
+  '"April-ANN v" APRILANN_VERSION_MAJOR "." APRILANN_VERSION_MINOR "." APRILANN_VERSION_RELEASE "-beta COMMIT " TOSTRING(GIT_COMMIT) "  Copyright (C) 2012-2013 DSIC-UPV, CEU-UCH"',
+  '"This program comes with ABSOLUTELY NO WARRANTY; for details see LICENSE.txt.\\nThis is free software, and you are welcome to redistribute it\\nunder certain conditions; see LICENSE.txt for details."',
+}

EXAMPLES/digits.lua

@@ -53,10 +53,13 @@ trainer:randomize_weights{
   inf         = -1,
   sup         =  1,
 }
+trainer:set_option("learning_rate", 0.01)
+trainer:set_option("momentum",      0.01)
+trainer:set_option("weight_decay",  1e-05)
+-- bias has weight_decay of ZERO
+trainer:set_layerwise_option("b.", "weight_decay", 0)
+
 trainer:save("jarl.net", "binary")
-thenet:set_option("learning_rate", 0.01)
-thenet:set_option("momentum",      0.01)
-thenet:set_option("weight_decay",  1e-05)
 
 training_data = {
   input_dataset  = train_input,

EXAMPLES/xor.lua

@@ -7,9 +7,10 @@ trainer:randomize_weights{
   inf        = -0.1,
   sup        =  0.1 }
 
-thenet:set_option("learning_rate", 8.0)
-thenet:set_option("momentum",      0.5)
-thenet:set_option("weight_decay",  1e-05)
+trainer:set_option("learning_rate", 8.0)
+trainer:set_option("momentum",      0.5)
+trainer:set_option("weight_decay",  1e-05)
+trainer:set_layerwise_option("b.*", "weight_decay", 0.0)
 
 m_xor = matrix.fromString[[
     4 3

README.md

@@ -78,8 +78,61 @@ You need to especify the `-I` option to the compiler, and all the extra_libs stu
 Exists one build file for each possible target: build_release.lua, build_debug.lua, build_mkl_release.lua,
 build_mkl_debug.lua, ... and so on.
 
+The binary will be generated at `bin/april-ann`, which incorporates the Lua 5.2
+interpreter and works without any dependency in Lua.  Besides, a shared library
+will be generated at `lib/aprilann.so`, so it is possible to use `require` from
+Lua to load April-ANN in a standard Lua 5.2 interpreter.
+
 ENJOY!
 
+Installation
+------------
+
+The installation is done executing:
+
+```
+$ sudo make install
+```
+
+This procedure copies the binary to system location in `/usr` (or in
+`/opt/local` for Mac OS X via MacPorts). The shared library is copied
+to Lua default directory, in order to load it by using `require` function.
+
+Use
+---
+
+- You can execute the standalone binary:
+
+```
+$ april-ann
+April-ANN v0.2.1-beta COMMIT 920  Copyright (C) 2012-2013 DSIC-UPV, CEU-UCH
+This program comes with ABSOLUTELY NO WARRANTY; for details see LICENSE.txt.
+This is free software, and you are welcome to redistribute it
+under certain conditions; see LICENSE.txt for details.
+Lua 5.2.2  Copyright (C) 1994-2013 Lua.org, PUC-Rio
+> print "Hello World!"
+Hello World!
+```
+
+- It is possible to use April-ANN as a Lua module, loading only the packages
+  which you need (i.e. `require("aprilann.matrix")`), or loading the full
+  library (`require("aprilann")`). **Be careful**, the April-ANN modules doesn't
+  follow Lua guidelines and have lateral effects because of the declaration of
+  tables, functions, and other values at the GLOBALs Lua table:
+
+```
+$ lua
+Lua 5.2.2  Copyright (C) 1994-2013 Lua.org, PUC-Rio
+> require "aprilann.matrix"
+> require "aprilann"
+April-ANN v0.2.1-beta COMMIT 920  Copyright (C) 2012-2013 DSIC-UPV, CEU-UCH
+This program comes with ABSOLUTELY NO WARRANTY; for details see LICENSE.txt.
+This is free software, and you are welcome to redistribute it
+under certain conditions; see LICENSE.txt for details.
+> print "Hello World!"
+Hello World!
+```
+
 Citation
 --------
 
@@ -123,7 +176,7 @@ Includes these sources
 - MersenneTwister: http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html
 - Median filter from Simon Perreault: http://nomis80.org/ctmf.html
 - RuningStat class for efficient and stable computation of mean and variance: http://www.johndcook.com/standard_deviation.html
-
+- Lua autocompletion rlcompleter release 2, by rthomas: https://github.com/rrthomas/lua-rlcompleter
 
 Wiki documentation
 ------------------

TEST/digitos/learndigits.lua

@@ -72,6 +72,13 @@ thenet  = ann.mlp.all_all.generate(description)
 trainer = trainable.supervised_trainer(thenet,
 				       ann.loss.multi_class_cross_entropy(10))
 trainer:build()
+
+trainer:set_option("learning_rate", 0.04)
+trainer:set_option("momentum",      0.02)
+trainer:set_option("weight_decay",  1e-05)
+-- bias has weight_decay of ZERO
+trainer:set_layerwise_option("b.", "weight_decay", 0)
+
 trainer:randomize_weights{
   random = aleat,
   inf    = inf,
@@ -126,10 +133,6 @@ datosvalidar = {
   bunch_size     = bunch_size,
 }
 
-thenet:set_option("learning_rate", 0.04)
-thenet:set_option("momentum",      0.02)
-thenet:set_option("weight_decay",  1e-05)
-
 -- datos para guardar una matriz de salida de la red
 msalida = matrix(200,10)
 dsalida = dataset.matrix(msalida, {

TEST/digitos/test.lua

@@ -74,13 +74,17 @@ val_output   = dataset.matrix(m2,
 
 thenet = ann.mlp.all_all.generate(description)
 if util.is_cuda_available() then thenet:set_use_cuda(true) end
-thenet:set_option("learning_rate", learning_rate)
-thenet:set_option("momentum",      momentum)
-thenet:set_option("weight_decay",  weight_decay)
 trainer = trainable.supervised_trainer(thenet,
 				       ann.loss.multi_class_cross_entropy(10),
 				       bunch_size)
 trainer:build()
+
+trainer:set_option("learning_rate", learning_rate)
+trainer:set_option("momentum",      momentum)
+trainer:set_option("weight_decay",  weight_decay)
+-- bias has weight_decay of ZERO
+trainer:set_layerwise_option("b.", "weight_decay", 0)
+
 trainer:randomize_weights{
   random      = weights_random,
   inf         = inf,

TEST/xor/xor-with-sparse-input.lua

@@ -41,10 +41,10 @@ end
 -----------------------------------------------------------
 
 net_component=ann.mlp.all_all.generate("2 inputs 2 logistic 1 logistic")
-net_component:set_option("learning_rate", learning_rate)
-net_component:set_option("momentum",      momentum)
-net_component:set_option("weight_decay",  weight_decay)
 trainer=trainable.supervised_trainer(net_component)
+trainer:set_option("learning_rate", learning_rate)
+trainer:set_option("momentum",      momentum)
+trainer:set_option("weight_decay",  weight_decay)
 trainer:build()
 trainer:set_loss_function(ann.loss.mse(net_component:get_output_size()))
 load_initial_weights(trainer.weights_table)

TEST/xor/xor.lua

@@ -39,11 +39,12 @@ end
 -----------------------------------------------------------
 
 net_component=ann.mlp.all_all.generate("2 inputs 2 logistic 1 logistic")
-net_component:set_option("learning_rate", learning_rate)
-net_component:set_option("momentum",      momentum)
-net_component:set_option("weight_decay",  weight_decay)
 trainer=trainable.supervised_trainer(net_component)
 trainer:build()
+trainer:set_option("learning_rate", learning_rate)
+trainer:set_option("momentum",      momentum)
+trainer:set_option("weight_decay",  weight_decay)
+trainer:set_layerwise_option("b.*", "weight_decay", 0.0)
 trainer:set_loss_function(ann.loss.mse(net_component:get_output_size()))
 load_initial_weights(trainer.weights_table)
 
@@ -135,6 +136,10 @@ net_component:pop()
 net_component:push( ann.components.actf.log_logistic() )
 trainer=trainable.supervised_trainer(net_component)
 trainer:build()
+trainer:set_option("learning_rate", learning_rate)
+trainer:set_option("momentum",      momentum)
+trainer:set_option("weight_decay",  weight_decay)
+trainer:set_layerwise_option("b.*", "weight_decay", 0.0)
 trainer:set_loss_function(ann.loss.cross_entropy(net_component:get_output_size()))
 trainer:save("ll.net", "ascii")
 load_initial_weights(trainer.weights_table)

VERSION.lua

@@ -0,0 +1,5 @@
+return {
+  '-DAPRILANN_VERSION_MAJOR=\'"0"\'',
+  '-DAPRILANN_VERSION_MINOR=\'"3"\'',
+  '-DAPRILANN_VERSION_RELEASE=\'"0"\'',
+}