Merge branch 'master' into Update_release

Author: zsdonghao

.circleci/config.yml

@@ -4,6 +4,7 @@ jobs:
 
   ###################################################################################
   #  TEST BUILDS with TensorLayer installed from Source - NOT PUSHED TO DOCKER HUB  #
+  #  [email protected]                                                          #
   ###################################################################################
 
   test_sources_py2_cpu:

.codacy.yaml

@@ -2,7 +2,7 @@
 ---
 engines:
   bandit:
-    enabled: false # FIXME: make it work
+    enabled: false # FIXME: make it works
 exclude_paths:
 - scripts/*
 - setup.py

README.md

@@ -145,7 +145,7 @@ The following table shows the training speeds of [VGG16](http://www.robots.ox.ac
 |   Mode    |       Lib       |  Data Format  | Max GPU Memory Usage(MB)  |Max CPU Memory Usage(MB) | Avg CPU Memory Usage(MB) | Runtime (sec) |
 | :-------: | :-------------: | :-----------: | :-----------------: | :-----------------: | :-----------------: | :-----------: |
 | AutoGraph | TensorFlow 2.0  | channel last  | 11833 |      2161         |        2136         |      74       |
-|           | Tensorlayer 2.0 | channel last  | 11833 |      2187         |        2169         |      76       |
+|           | TensorLayer 2.0 | channel last  | 11833 |      2187         |        2169         |      76       |
 |   Graph   |      Keras      | channel last  | 8677 |      2580         |        2576         |      101       |
 |   Eager   | TensorFlow 2.0  | channel last  | 8723 |      2052         |        2024         |      97       |
 |           | TensorLayer 2.0 | channel last  | 8723 |      2010         |        2007         |      95       |

README.rst

@@ -17,52 +17,20 @@ to build real-world AI applications. TensorLayer is awarded the 2017
 Best Open Source Software by the `ACM Multimedia
 Society <http://www.acmmm.org/2017/mm-2017-awardees/>`__.
 
-Why another deep learning library: TensorLayer
-==============================================
-
-As deep learning practitioners, we have been looking for a library that
-can address various development purposes. This library is easy to adopt
-by providing diverse examples, tutorials and pre-trained models. Also,
-it allow users to easily fine-tune TensorFlow; while being suitable for
-production deployment. TensorLayer aims to satisfy all these purposes.
-It has three key features:
-
--  **Simplicity** : TensorLayer lifts the low-level dataflow interface
-   of TensorFlow to *high-level* layers / models. It is very easy to
-   learn through the rich `example
-   codes <https://github.com/tensorlayer/awesome-tensorlayer>`__
-   contributed by a wide community.
--  **Flexibility** : TensorLayer APIs are transparent: it does not
-   mask TensorFlow from users; but leaving massive hooks that help
-   *low-level tuning* and *deep customization*.
--  **Zero-cost Abstraction** : TensorLayer can achieve the *full
-   power* of TensorFlow. The following table shows the training speeds
-   of classic models using TensorLayer and native TensorFlow on a Titan
-   X Pascal GPU.
-
-   +---------------+-----------------+-----------------+-----------------+
-   |               | CIFAR-10        | PTB LSTM        | Word2Vec        |
-   +===============+=================+=================+=================+
-   | TensorLayer   | 2528 images/s   | 18063 words/s   | 58167 words/s   |
-   +---------------+-----------------+-----------------+-----------------+
-   | TensorFlow    | 2530 images/s   | 18075 words/s   | 58181 words/s   |
-   +---------------+-----------------+-----------------+-----------------+
-
-TensorLayer stands at a unique spot in the library landscape. Other
-wrapper libraries like Keras and TFLearn also provide high-level
-abstractions. They, however, often hide the underlying engine from
-users, which make them hard to customize and fine-tune. On the contrary,
-TensorLayer APIs are generally flexible and transparent. Users often
-find it easy to start with the examples and tutorials, and then dive
-into TensorFlow seamlessly. In addition, TensorLayer does not create
-library lock-in through native supports for importing components from
-Keras, TFSlim and TFLearn.
-
-TensorLayer has a fast growing usage among top researchers and
-engineers, from universities like Imperial College London, UC Berkeley,
-Carnegie Mellon University, Stanford University, and University of
-Technology of Compiegne (UTC), and companies like Google, Microsoft,
-Alibaba, Tencent, Xiaomi, and Bloomberg.
+Design Features
+=================
+
+TensorLayer is a new deep learning library designed with simplicity, flexibility and high-performance in mind.
+
+- **Simplicity** : TensorLayer has a high-level layer/model abstraction which is effortless to learn. You can learn how deep learning can benefit your AI tasks in minutes through the massive [examples](https://github.com/tensorlayer/awesome-tensorlayer).
+- **Flexibility** : TensorLayer APIs are transparent and flexible, inspired by the emerging PyTorch library. Compared to the Keras abstraction, TensorLayer makes it much easier to build and train complex AI models.
+- **Zero-cost Abstraction** : Though simple to use, TensorLayer does not require you to make any compromise in the performance of TensorFlow (Check the following benchmark section for more details).
+
+TensorLayer stands at a unique spot in the TensorFlow wrappers. Other wrappers like Keras and TFLearn
+hide many powerful features of TensorFlow and provide little support for writing custom AI models. Inspired by PyTorch, TensorLayer APIs are simple, flexible and Pythonic,
+making it easy to learn while being flexible enough to cope with complex AI tasks.
+TensorLayer has a fast-growing community. It has been used by researchers and engineers all over the world, including those from  Peking University,
+Imperial College London, UC Berkeley, Carnegie Mellon University, Stanford University, and companies like Google, Microsoft, Alibaba, Tencent, Xiaomi, and Bloomberg.
 
 Install
 =======

docs/modules/activation.rst

@@ -35,6 +35,7 @@ For more complex activation, TensorFlow API will be required.
    sign
    hard_tanh
    pixel_wise_softmax
+   mish
 
 Ramp
 ------
@@ -68,6 +69,10 @@ Pixel-wise softmax
 --------------------
 .. autofunction:: pixel_wise_softmax
 
+mish
+---------
+.. autofunction:: mish
+
 Parametric activation
 ------------------------------
 See ``tensorlayer.layers``.

docs/user/contributing.rst

@@ -40,8 +40,10 @@ For TensorLayer 1.x, it was actively developed and maintained by the following p
 - **Hao Dong** (`@zsdonghao <https://github.com/zsdonghao>`_) - `<https://zsdonghao.github.io>`_
 - **Jonathan Dekhtiar** (`@DEKHTIARJonathan <https://github.com/DEKHTIARJonathan>`_) - `<https://www.jonathandekhtiar.eu>`_
 - **Luo Mai** (`@luomai <https://github.com/luomai>`_) - `<http://www.doc.ic.ac.uk/~lm111/>`_
+- **Pan Wang** (`@FerociousPanda <http://github.com/FerociousPanda>`_) - `<http://github.com/FerociousPanda>`_  (UI)
 - **Simiao Yu** (`@nebulaV <https://github.com/nebulaV>`_) - `<https://nebulav.github.io>`_
 
+
 Numerous other contributors can be found in the `Github Contribution Graph <https://github.com/tensorlayer/tensorlayer/graphs/contributors>`_.
 
 

examples/pretrained_cnn/tutorial_load_ckpt_weights_to_tensorlayer.py

@@ -0,0 +1,70 @@
+#! /usr/bin/python
+# -*- coding: utf-8 -*-
+
+import tensorlayer as tl
+from tensorlayer.layers import (Input, Conv2d, Flatten, Dense, MaxPool2d)
+from tensorlayer.models import Model
+from tensorlayer.files import maybe_download_and_extract
+import numpy as np
+import tensorflow as tf
+
+filename = 'ckpt_parameters.zip'
+url_score = 'https://media.githubusercontent.com/media/tensorlayer/pretrained-models/master/models/'
+
+# download weights
+down_file = tl.files.maybe_download_and_extract(
+    filename=filename, working_directory='model/', url_source=url_score, extract=True
+)
+
+model_file = 'model/ckpt_parameters'
+
+# ckpt to npz, rename_key used to match TL naming rule
+tl.files.ckpt_to_npz_dict(model_file, rename_key=True)
+weights = np.load('model.npz', allow_pickle=True)
+
+# View the parameters and weights shape
+for key in weights.keys():
+    print(key, weights[key].shape)
+
+
+# build model
+def create_model(inputs_shape):
+    W_init = tl.initializers.truncated_normal(stddev=5e-2)
+    W_init2 = tl.initializers.truncated_normal(stddev=0.04)
+    ni = Input(inputs_shape)
+    nn = Conv2d(64, (3, 3), (1, 1), padding='SAME', act=tf.nn.relu, W_init=W_init, name='conv1_1')(ni)
+    nn = MaxPool2d((2, 2), (2, 2), padding='SAME', name='pool1_1')(nn)
+    nn = Conv2d(64, (3, 3), (1, 1), padding='SAME', act=tf.nn.relu, W_init=W_init, b_init=None, name='conv1_2')(nn)
+    nn = MaxPool2d((2, 2), (2, 2), padding='SAME', name='pool1_2')(nn)
+
+    nn = Conv2d(128, (3, 3), (1, 1), padding='SAME', act=tf.nn.relu, W_init=W_init, b_init=None, name='conv2_1')(nn)
+    nn = MaxPool2d((2, 2), (2, 2), padding='SAME', name='pool2_1')(nn)
+    nn = Conv2d(128, (3, 3), (1, 1), padding='SAME', act=tf.nn.relu, W_init=W_init, b_init=None, name='conv2_2')(nn)
+    nn = MaxPool2d((2, 2), (2, 2), padding='SAME', name='pool2_2')(nn)
+
+    nn = Conv2d(256, (3, 3), (1, 1), padding='SAME', act=tf.nn.relu, W_init=W_init, b_init=None, name='conv3_1')(nn)
+    nn = MaxPool2d((2, 2), (2, 2), padding='SAME', name='pool3_1')(nn)
+    nn = Conv2d(256, (3, 3), (1, 1), padding='SAME', act=tf.nn.relu, W_init=W_init, b_init=None, name='conv3_2')(nn)
+    nn = MaxPool2d((2, 2), (2, 2), padding='SAME', name='pool3_2')(nn)
+
+    nn = Conv2d(512, (3, 3), (1, 1), padding='SAME', act=tf.nn.relu, W_init=W_init, b_init=None, name='conv4_1')(nn)
+    nn = MaxPool2d((2, 2), (2, 2), padding='SAME', name='pool4_1')(nn)
+    nn = Conv2d(512, (3, 3), (1, 1), padding='SAME', act=tf.nn.relu, W_init=W_init, b_init=None, name='conv4_2')(nn)
+    nn = MaxPool2d((2, 2), (2, 2), padding='SAME', name='pool4_2')(nn)
+
+    nn = Flatten(name='flatten')(nn)
+    nn = Dense(1000, act=None, W_init=W_init2, name='output')(nn)
+
+    M = Model(inputs=ni, outputs=nn, name='cnn')
+    return M
+
+
+net = create_model([None, 224, 224, 3])
+# loaded weights whose name is not found in network's weights will be skipped.
+# If ckpt has the same naming rule as TL, We can restore the model with tl.files.load_and_assign_ckpt(model_dir=, network=, skip=True)
+tl.files.load_and_assign_npz_dict(network=net, skip=True)
+
+# you can use the following code to view the restore the model parameters.
+net_weights_name = [w.name for w in net.all_weights]
+for i in range(len(net_weights_name)):
+    print(net_weights_name[i], net.all_weights[net_weights_name.index(net_weights_name[i])])

examples/reinforcement_learning/tutorial_atari_pong.py

@@ -7,14 +7,15 @@
 Pixels” which is a minimalistic implementation of deep reinforcement learning by
 using python-numpy and OpenAI gym environment.
 The code here is the reimplementation of Karpathy's Blog by using TensorLayer.
-Compare with Karpathy's code, we store observation for a batch, he store
-observation for a episode only, they store gradients instead. (so we will use
+Compare with Karpathy's code, we store observation for a batch, but he store
+observation for only one episode and gradients. (so we will use
 more memory if the observation is very large.)
-FEEL FREE TO JOIN US !
+
 TODO
 -----
 - update grads every step rather than storing all observation!
 - [email protected]
+
 References
 ------------
 - http://karpathy.github.io/2016/05/31/rl/

setup.cfg

@@ -23,7 +23,7 @@ based_on_style=google
 # The number of columns to use for indentation.
 indent_width = 4
 
-# The column limit.
+# The column limit. (larger than usual)
 column_limit=120
 
 # Place each dictionary entry onto its own line.
@@ -76,4 +76,4 @@ no_spaces_around_selected_binary_operators = True
 allow_multiline_lambdas = True
 
 SPLIT_PENALTY_FOR_ADDED_LINE_SPLIT = 10
-SPLIT_PENALTY_AFTER_OPENING_BRACKET = 500
+SPLIT_PENALTY_AFTER_OPENING_BRACKET = 500

tensorlayer/activation.py

@@ -19,6 +19,7 @@
     'htanh',
     'hard_tanh',
     'pixel_wise_softmax',
+    'mish',
 ]
 
 
@@ -339,6 +340,25 @@ def pixel_wise_softmax(x, name='pixel_wise_softmax'):
         return tf.nn.softmax(x)
 
 
+def mish(x):
+    """Mish activation function.
+
+    Reference: [Mish: A Self Regularized Non-Monotonic Neural Activation Function .Diganta Misra, 2019]<https://arxiv.org/abs/1908.08681>
+
+    Parameters
+    ----------
+    x : Tensor
+        input.
+
+    Returns
+    -------
+    Tensor
+        A ``Tensor`` in the same type as ``x``.
+
+    """
+    return x * tf.math.tanh(tf.math.softplus(x))
+
+
 # Alias
 lrelu = leaky_relu
 lrelu6 = leaky_relu6