Docker Integration (#648)

Jonathan DEKHTIAR · lgarithm · commit 8e2416fe2a04 · 2018-05-27T08:30:13.000+08:00
* no message

* Configuration CircleCI fixed

* Python added

* Config Fix

* CircleCI test fix

* Readme Updated

* CircleCI conditional build added

* Error Fix

* Error fix

* Changelog Updated

* CircleCI config updated with pinned version

* Missing Package added "requests"

* CircleCI conf updated

* CircleCI updated

* CircleCI test

* Test CircleCI

* Test CircleCI

* Circle CI TEST

* Circle CI Build Test

* Circle BUILD TEST

* problematic line removed

* problematic line removed

* DockerFiles Fixed

* double quote fix

* Test CircleCI

* Docker Push Re-Organised

* Test Workflow CircleCI

* Code Checkout Added

* Test CircleCI release Dev

* Version Prefix fix

* Last Version Build Restored

* Nightly Builds added

* Nightly build test

* CircleCI - Runs Name Changed

* Nightly builds errors fix

* Test still OK CircleCI

* Test CircleCI still OK

* Nightly builds added

* CircleCI config Updated

* Renamed test builds

* CircleCI Approval Push Added on Tag

* Approval Test

* Test Approval

* CircleCI Config Cleaned for Merge
diff --git a/.circleci/config.yml b/.circleci/config.yml
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -71,26 +71,30 @@ To release a new version, please update the changelog as followed:
 ### Added
 - API:
   - `tl.alphas` and `tl.alphas_like` added following the tf.ones/zeros and tf.zeros_like/ones_like (by @DEKHTIARJonathan in #580)
-- Tutorials:
-  - `tutorial_tfslim` has been introduced to show how to use `SlimNetsLayer` (by @2wins in #560).
-- Test:
-  - `test_utils_predict.py` added to reproduce and fix issue #288 (by @2wins in #566)
-  - `Layer_DeformableConvolution_Test` added to reproduce issue #572 with deformable convolution (by @DEKHTIARJonathan in #573)
-  - `Array_Op_Alphas_Test` and `Array_Op_Alphas_Like_Test` added to test `tensorlayer/array_ops.py` file (by @DEKHTIARJonathan in #580)
-  - `test_optimizer_amsgrad.py` added to test `AMSGrad` optimizer (by @DEKHTIARJonathan in #636)
-  - `test_logging.py` added to insure robustness of the logging API (by @DEKHTIARJonathan in #645)
 - CI Tool:
   - [Stale Probot](https://github.com/probot/stale) added to clean stale issues (by @DEKHTIARJonathan in #573)
   - [Changelog Probot](https://github.com/mikz/probot-changelog) Configuration added (by @DEKHTIARJonathan in #637)
   - Travis Builds now handling a matrix of TF Version from TF==1.6.0 to TF==1.8.0 (by @DEKHTIARJonathan in #644)
-- Layer:
-  - ElementwiseLambdaLayer added to use custom function to connect multiple layer inputs (by @One-sixth in #579)
+  - CircleCI added to build and upload Docker Containers for each PR merged and tag release (by @DEKHTIARJonathan in #648)
+- Docker:
+  - Containers for each release and for each PR merged on master built (by @DEKHTIARJonathan in #648)
+  - Containers built in the following configurations: py2+cpu, py2+gpu, py3+cpu, and py3+gpu (by @DEKHTIARJonathan in #648)
 - Documentation:
   - Release semantic version added on index page (by @DEKHTIARJonathan in #633)
   - Optimizers page added (by @DEKHTIARJonathan in #636)
   - `AMSGrad` added on Optimizers page added (by @DEKHTIARJonathan in #636)
+- Layer:
+  - ElementwiseLambdaLayer added to use custom function to connect multiple layer inputs (by @One-sixth in #579)
 - Optimizer:
   - AMSGrad Optimizer added based on `On the Convergence of Adam and Beyond (ICLR 2018)` (by @DEKHTIARJonathan in #636)
+- Test:
+  - `test_utils_predict.py` added to reproduce and fix issue #288 (by @2wins in #566)
+  - `Layer_DeformableConvolution_Test` added to reproduce issue #572 with deformable convolution (by @DEKHTIARJonathan in #573)
+  - `Array_Op_Alphas_Test` and `Array_Op_Alphas_Like_Test` added to test `tensorlayer/array_ops.py` file (by @DEKHTIARJonathan in #580)
+  - `test_optimizer_amsgrad.py` added to test `AMSGrad` optimizer (by @DEKHTIARJonathan in #636)
+  - `test_logging.py` added to insure robustness of the logging API (by @DEKHTIARJonathan in #645)
+- Tutorials:
+  - `tutorial_tfslim` has been introduced to show how to use `SlimNetsLayer` (by @2wins in #560).
 
 ### Changed
 - Tensorflow CPU & GPU dependencies moved to separated requirement files in order to allow PyUP.io to parse them (by @DEKHTIARJonathan in #573)
diff --git a/Dockerfile b/Dockerfile
diff --git a/Dockerfile.gpu b/Dockerfile.gpu
diff --git a/README.md b/README.md
@@ -13,15 +13,16 @@
 </div>
 </a>
 
-[![Build Status](https://travis-ci.org/tensorlayer/tensorlayer.svg?branch=master)](https://travis-ci.org/tensorlayer/tensorlayer)
+[![Build Status](https://img.shields.io/travis/tensorlayer/tensorlayer.svg?label=Travis&branch=master)](https://travis-ci.org/tensorlayer/tensorlayer)
 [![PyPI version](https://badge.fury.io/py/tensorlayer.svg)](https://pypi.org/project/tensorlayer/)
 [![Github commits (since latest release)](https://img.shields.io/github/commits-since/tensorlayer/tensorlayer/latest.svg)](https://github.com/tensorlayer/tensorlayer/compare/1.8.5...master)
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/tensorlayer.svg)](https://pypi.org/project/tensorlayer/)
 [![Supported TF Version](https://img.shields.io/badge/tensorflow-1.6.0+-blue.svg)](https://github.com/tensorflow/tensorflow/releases)
 [![Codacy Badge](https://api.codacy.com/project/badge/Grade/ca2a29ddcf7445588beff50bee5406d9)](https://app.codacy.com/app/tensorlayer/tensorlayer)
-[![Docker Pulls](https://img.shields.io/docker/pulls/tensorlayer/tensorlayer.svg?maxAge=604800)](https://hub.docker.com/r/tensorlayer/tensorlayer/)
 
-[![Documentation Status](https://readthedocs.org/projects/tensorlayer/badge/?version=stable)](https://tensorlayer.readthedocs.io/)
+[![CircleCI](https://img.shields.io/circleci/project/github/tensorlayer/tensorlayer.svg?label=Docker%20Build&branch=master)](https://circleci.com/gh/tensorlayer/tensorlayer/tree/master)
+[![Docker Pulls](https://img.shields.io/docker/pulls/tensorlayer/tensorlayer.svg?maxAge=604800)](https://hub.docker.com/r/tensorlayer/tensorlayer/)
+[![Documentation Status](https://img.shields.io/readthedocs/tensorlayer/stable.svg?label=ReadTheDocs)](https://tensorlayer.readthedocs.io/)
 [![PyUP Updates](https://pyup.io/repos/github/tensorlayer/tensorlayer/shield.svg)](https://pyup.io/repos/github/tensorlayer/tensorlayer/)
 [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/tensorlayer/Lobby)
 
diff --git a/docker/DockerLint.bat b/docker/DockerLint.bat
@@ -0,0 +1,8 @@
+docker pull hadolint/hadolint:latest
+
+docker run --rm -i hadolint/hadolint hadolint --ignore DL3007 - < python2\cpu\Dockerfile
+docker run --rm -i hadolint/hadolint hadolint --ignore DL3007 - < python2\gpu\Dockerfile
+docker run --rm -i hadolint/hadolint hadolint --ignore DL3007 - < python3\cpu\Dockerfile
+docker run --rm -i hadolint/hadolint hadolint --ignore DL3007 - < python3\gpu\Dockerfile
+
+PAUSE;
diff --git a/docker/pypi_list.py b/docker/pypi_list.py
@@ -0,0 +1,68 @@
+import argparse
+import requests
+import logging
+
+import pip._internal
+
+if __name__ == "__main__":
+
+    parser = argparse.ArgumentParser(description='Get the nth version of a given package')
+    parser.add_argument('--package', type=str, required=True, help='The PyPI you want to inspect')
+    parser.add_argument('--nth_last_version', type=int, default=1, help='The nth last package will be retrieved')
+    parser.add_argument('--prerelease', help='Get PreRelease Package Version', action='store_true')
+    parser.add_argument('--debug', help='Print debug information', action='store_true')
+
+    args = parser.parse_args()
+
+    # create logger
+    logger = logging.getLogger("PyPI_CLI")
+
+    formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
+
+    ch = logging.StreamHandler()
+    ch.setLevel(logging.DEBUG)
+    ch.setFormatter(formatter)
+    logger.addHandler(ch)
+
+    if args.debug:
+        logger.setLevel(logging.DEBUG)
+
+    logger.debug("Package: %s" % args.package)
+    logger.debug("nth_last_version: %s" % args.nth_last_version)
+    logger.debug("prerelease: %s" % args.prerelease)
+    logger.debug("debug: %s" % args.debug)
+
+    finder = pip._internal.index.PackageFinder(
+        [],
+        ['https://pypi.python.org/simple'],
+        session=requests.Session()
+    )
+    results = finder.find_all_candidates(args.package)
+    tmp_versions = [str(p.version) for p in results]
+
+    logger.debug("%s" % tmp_versions)
+
+    versions = list()
+    for el in tmp_versions:
+        if el not in versions:
+            versions.append(el)
+
+    pos = -1
+    nth_version = 1
+
+    while True:
+        fetched_version = versions[pos]
+
+        logger.debug("Version: %s" % fetched_version)
+
+        if nth_version == args.nth_last_version:
+            if args.prerelease or not ("rc" in fetched_version or "a" in fetched_version or "b" in fetched_version):
+                break
+            else:
+                pos -= 1
+                continue
+
+        pos -= 1
+        nth_version += 1
+
+    print(fetched_version)
diff --git a/docker/python2/cpu/Dockerfile b/docker/python2/cpu/Dockerfile
@@ -0,0 +1,20 @@
+FROM tensorflow/tensorflow:latest
+
+LABEL version="1.0" maintainer="Jonathan DEKHTIAR <contact@jonathandekhtiar.eu>"
+
+ARG TL_VERSION
+
+RUN if [ -z "$TL_VERSION" ]; then \
+        echo "Building a Nightly Release" \
+        && apt-get update \
+        && apt-get install -y git \
+        && mkdir /tensorlayer_dist/ \
+        && cd /tensorlayer_dist/ \
+        && git clone https://github.com/tensorlayer/tensorlayer.git \
+        && cd tensorlayer \
+        && pip install -e .[db,dev,test,doc] \
+        && rm -rf /var/lib/apt/lists/* ; \
+    else \
+        echo "Building Tag Release:" "$TL_VERSION" \
+        && pip install tensorlayer=="$TL_VERSION" ; \
+    fi
diff --git a/docker/python2/gpu/Dockerfile b/docker/python2/gpu/Dockerfile
@@ -0,0 +1,20 @@
+FROM tensorflow/tensorflow:latest-gpu
+
+LABEL version="1.0" maintainer="Jonathan DEKHTIAR <contact@jonathandekhtiar.eu>"
+
+ARG TL_VERSION
+
+RUN if [ -z "$TL_VERSION" ]; then \
+        echo "Building a Nightly Release" \
+        && apt-get update \
+        && apt-get install -y git \
+        && mkdir /tensorlayer_dist/ \
+        && cd /tensorlayer_dist/ \
+        && git clone https://github.com/tensorlayer/tensorlayer.git \
+        && cd tensorlayer \
+        && pip install -e .[db,dev,test,doc] \
+        && rm -rf /var/lib/apt/lists/* ; \
+    else \
+        echo "Building Tag Release:" "$TL_VERSION" \
+        && pip install tensorlayer=="$TL_VERSION" ; \
+    fi
diff --git a/docker/python3/cpu/Dockerfile b/docker/python3/cpu/Dockerfile
@@ -0,0 +1,20 @@
+FROM tensorflow/tensorflow:latest-py3
+
+LABEL version="1.0" maintainer="Jonathan DEKHTIAR <contact@jonathandekhtiar.eu>"
+
+ARG TL_VERSION
+
+RUN if [ -z "$TL_VERSION" ]; then \
+        echo "Building a Nightly Release" \
+        && apt-get update \
+        && apt-get install -y git \
+        && mkdir /tensorlayer_dist/ \
+        && cd /tensorlayer_dist/ \
+        && git clone https://github.com/tensorlayer/tensorlayer.git \
+        && cd tensorlayer \
+        && pip install -e .[db,dev,test,doc] \
+        && rm -rf /var/lib/apt/lists/* ; \
+    else \
+        echo "Building Tag Release:" "$TL_VERSION" \
+        && pip install tensorlayer=="$TL_VERSION" ; \
+    fi
diff --git a/docker/python3/gpu/Dockerfile b/docker/python3/gpu/Dockerfile
@@ -0,0 +1,20 @@
+FROM tensorflow/tensorflow:latest-gpu-py3
+
+LABEL version="1.0" maintainer="Jonathan DEKHTIAR <contact@jonathandekhtiar.eu>"
+
+ARG TL_VERSION
+
+RUN if [ -z "$TL_VERSION" ]; then \
+        echo "Building a Nightly Release" \
+        && apt-get update \
+        && apt-get install -y git \
+        && mkdir /tensorlayer_dist/ \
+        && cd /tensorlayer_dist/ \
+        && git clone https://github.com/tensorlayer/tensorlayer.git \
+        && cd tensorlayer \
+        && pip install -e .[db,dev,test,doc] \
+        && rm -rf /var/lib/apt/lists/* ; \
+    else \
+        echo "Building Tag Release:" "$TL_VERSION" \
+        && pip install tensorlayer=="$TL_VERSION" ; \
+    fi
diff --git a/docker/version_prefix.py b/docker/version_prefix.py
@@ -0,0 +1,37 @@
+import argparse
+import logging
+
+if __name__ == "__main__":
+
+    parser = argparse.ArgumentParser(description='Determine the version prefix to apply depending on the version name')
+
+    parser.add_argument(
+        '--version',
+        type=str,
+        required=True,
+        help='The Package Version to be installed in the container'
+    )
+
+    parser.add_argument('--debug', help='Print debug information', action='store_true')
+
+    args = parser.parse_args()
+
+    # create logger
+    logger = logging.getLogger("VERSION_PREFIX_CLI")
+
+    formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
+
+    ch = logging.StreamHandler()
+    ch.setLevel(logging.DEBUG)
+    ch.setFormatter(formatter)
+    logger.addHandler(ch)
+
+    if args.debug:
+        logger.setLevel(logging.DEBUG)
+
+    logger.debug("Package Version: %s" % args.version)
+
+    if "rc" in args.version or "a" in args.version or "b" in args.version:
+        print("latest-dev")
+    else:
+        print("latest")
