Merge pull request #2 from gjbex/development

Rework material

Author: gjbex

README.md

@@ -14,7 +14,7 @@ For information on the training, see the website
    illustrate concepts.
 1. [`environment.yml`](environment.yml): conda environment file intended to be
    cross-platform.
-1. [`python_for_data_science_linux64_conda_specs.txt`](python_for_data_science_linux64_conda_specs.txt):
+1. [`python_for_machine_learning_linux64_conda_specs.txt`](python_for_machine_learning_linux64_conda_specs.txt):
    conda environment specification file specific for 64-bit Linux to precisely
    reproduce the environment on which the code was developed.
 1. [License](LICENSE): license information for the material in this repository.

docs/README.md

@@ -0,0 +1,70 @@
+The Python programming language is increasingly popular.  It is a
+versatile language for general purpose programming and accessible
+for novice programmers.  However, it is also the de facto go-to
+language for machine learning applciations.  This training
+introduces modules that are useful in that context.
+
+
+## Learning outcomes
+
+When you complete this training you will
+
+  * understand what supervised and unsupervised learning are;
+  * understand the workflow required for supervised learning;
+  * be able to implement that workflow in scikit-learn;
+  * know how to use algorithms in scikit-learn such as ridge regression
+    and naive Bayes classification;
+  * be able to apply k-means clustering;
+  * understand the main concepts in deep neural networks;
+  * be able to apply a convolutional neural network (CNN) to an image
+    classification task;
+  * understand the concepts underpinning word embeddings;
+  * be able to use recurrent network architectures such as 
+    long short term memory (LSTM) to natural language problems;
+  * be aware of best practices and pitfalls in machine learning.
+
+
+## Schedule
+
+Total duration: 4 hours.
+
+  | Subject                                     | Duration |
+  |---------------------------------------------|----------|
+  | introduction and motivation                 | 20 min.  |
+  | scikit-learn: regression                    | 40 min.  |
+  | scikit-learn: classification                | 30 min.  |
+  | science-learn: clustering                   | 20 min.  |
+  | coffee break                                | 10 min.  |
+  | introduction to neural networks             | 10 min.  |
+  | Keras: image classification with CNNs       | 40 min.  |
+  | Keras: sentiment classification with LSTM   | 40 min.  |
+  | hyperparameter optimization                 | 30 min.  |
+  | wrap up                                     | 10 min.  |
+
+
+## Training materials
+
+Slides are available in the
+ [GitHub repository](https://github.com/gjbex/Python-for-machine-learning),
+as well as example code and hands-on material.
+
+
+## Target audience
+
+This training is for you if you need to use Python for machine learning
+pplcations.
+
+
+## Prerequisites
+
+You will need experience programming in Python.  This is not a training that starts
+from scratch.  Familiarity with numpy is not required, but would be beneficial.
+Familiarity with numpy, pandas and matplotlib is strongly recommended.
+
+If you plan to do Python programming in a Linux or HPC environment you should
+be familiar with these as well.
+
+
+## Trainer(s)
+
+  * Geert Jan Bex ([[email protected]](mailto:[email protected]))

environment.yml

@@ -0,0 +1,14 @@
+name: python_for_machine_learning
+channels:
+  - defaults
+dependencies:
+  - keras
+  - scikit-learn
+  - pandas
+  - jupyterlab
+  - nodejs
+  - matplotlib
+  - seaborn
+  - holoviews
+  - ipywidgets
+  - hyperopt

python_for_machine_learning.pptx

@@ -0,0 +1,153 @@
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+@EXPLICIT
+https://repo.anaconda.com/pkgs/main/linux-64/_libgcc_mutex-0.1-main.conda
+https://repo.anaconda.com/pkgs/main/linux-64/_tflow_select-2.3.0-mkl.conda
+https://repo.anaconda.com/pkgs/main/linux-64/blas-1.0-mkl.conda
+https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2019.11.28-hecc5488_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/intel-openmp-2019.4-243.conda
+https://repo.anaconda.com/pkgs/main/linux-64/ld_impl_linux-64-2.33.1-h53a641e_7.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libgfortran-ng-7.3.0-hdf63c60_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libstdcxx-ng-9.1.0-hdf63c60_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-9.1.0-hdf63c60_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/mkl-2019.4-243.conda
+https://repo.anaconda.com/pkgs/main/linux-64/c-ares-1.15.0-h7b6447c_1001.conda
+https://repo.anaconda.com/pkgs/main/linux-64/expat-2.2.6-he6710b0_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/gmp-6.1.2-h6c8ec71_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/icu-58.2-h9c2bf20_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/jpeg-9b-h024ee3a_2.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libffi-3.2.1-hd88cf55_4.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libsodium-1.0.16-h1bed415_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libuuid-1.0.3-h1bed415_2.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libxcb-1.13-h1bed415_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/ncurses-6.1-he6710b0_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/nodejs-10.13.0-he6710b0_0.conda
+https://conda.anaconda.org/conda-forge/linux-64/openssl-1.1.1d-h516909a_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pcre-8.43-he6710b0_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/xz-5.2.4-h14c3975_4.conda
+https://repo.anaconda.com/pkgs/main/linux-64/yaml-0.1.7-had09818_2.conda
+https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.11-h7b6447c_3.conda
+https://repo.anaconda.com/pkgs/main/linux-64/glib-2.63.1-h5a9c865_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/hdf5-1.10.4-hb1b8bf9_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libedit-3.1.20181209-hc058e9b_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libpng-1.6.37-hbc83047_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libprotobuf-3.11.2-hd408876_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libxml2-2.9.9-hea5a465_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pandoc-2.2.3.2-0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/readline-7.0-h7b6447c_5.conda
+https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.8-hbc83047_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/zeromq-4.3.1-he6710b0_3.conda
+https://repo.anaconda.com/pkgs/main/linux-64/zstd-1.3.7-h0b5b093_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/dbus-1.13.12-h746ee38_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/freetype-2.9.1-h8a8886c_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/gstreamer-1.14.0-hb453b48_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/libtiff-4.1.0-h2733197_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/sqlite-3.30.1-h7b6447c_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/fontconfig-2.13.0-h9420a91_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/gst-plugins-base-1.14.0-hbbd80ab_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/python-3.7.6-h0371630_2.conda
+https://repo.anaconda.com/pkgs/main/linux-64/astor-0.8.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/attrs-19.3.0-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/backcall-0.1.0-py37_0.conda
+https://conda.anaconda.org/conda-forge/linux-64/certifi-2019.11.28-py37_0.tar.bz2
+https://conda.anaconda.org/conda-forge/noarch/cloudpickle-1.2.2-py_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/decorator-4.4.1-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/defusedxml-0.6.0-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/entrypoints-0.3-py37_0.conda
+https://conda.anaconda.org/conda-forge/linux-64/future-0.18.2-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/gast-0.2.2-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/ipython_genutils-0.2.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/json5-0.8.5-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/kiwisolver-1.1.0-py37he6710b0_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/markupsafe-1.1.1-py37h7b6447c_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/mistune-0.8.4-py37h7b6447c_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/more-itertools-8.0.2-py_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/olefile-0.46-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pandocfilters-1.4.2-py37_1.conda
+https://repo.anaconda.com/pkgs/main/noarch/param-1.9.2-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/parso-0.5.2-py_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pickleshare-0.7.5-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/prometheus_client-0.7.1-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/ptyprocess-0.6.0-py37_0.conda
+https://conda.anaconda.org/conda-forge/linux-64/pymongo-3.10.1-py37he1b5a44_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/pyparsing-2.4.6-py_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/pytz-2019.3-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pyyaml-5.2-py37h7b6447c_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pyzmq-18.1.0-py37he6710b0_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/qt-5.9.7-h5867ecd_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/send2trash-1.5.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/sip-4.19.8-py37hf484d3e_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/six-1.13.0-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/termcolor-1.1.0-py37_1.conda
+https://repo.anaconda.com/pkgs/main/noarch/testpath-0.4.4-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/tornado-6.0.3-py37h7b6447c_0.conda
+https://conda.anaconda.org/conda-forge/noarch/tqdm-4.41.1-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/wcwidth-0.1.7-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/webencodings-0.5.1-py37_1.conda
+https://repo.anaconda.com/pkgs/main/noarch/werkzeug-0.16.0-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/wrapt-1.11.2-py37h7b6447c_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/absl-py-0.8.1-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/cycler-0.10.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/google-pasta-0.1.8-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jedi-0.15.2-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/mkl-service-2.3.0-py37he904b0f_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/packaging-20.0-py_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pexpect-4.7.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pillow-7.0.0-py37hb39fc2d_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pyqt-5.9.2-py37h05f1152_2.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pyrsistent-0.15.6-py37h7b6447c_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/python-dateutil-2.8.1-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/pyviz_comms-0.7.2-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/setuptools-44.0.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/terminado-0.8.3-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/traitlets-4.3.3-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/zipp-0.6.0-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/bleach-3.1.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/grpcio-1.16.1-py37hf8bcb03_1.conda
+https://repo.anaconda.com/pkgs/main/linux-64/importlib_metadata-1.3.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/jinja2-2.10.3-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/joblib-0.14.1-py_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/jupyter_core-4.6.1-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/markdown-3.1.1-py37_0.conda
+https://conda.anaconda.org/conda-forge/noarch/networkx-2.4-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/numpy-base-1.17.4-py37hde5b4d6_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/protobuf-3.11.2-py37he6710b0_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/pygments-2.5.2-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/wheel-0.33.6-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/jsonschema-3.2.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/jupyter_client-5.3.4-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/pip-19.3.1-py37_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/prompt_toolkit-3.0.2-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/ipython-7.11.1-py37h39e3cac_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/nbformat-4.4.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/ipykernel-5.1.3-py37h39e3cac_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/nbconvert-5.6.1-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/notebook-6.0.2-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/jupyterlab_server-1.0.6-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/widgetsnbextension-3.5.1-py37_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/ipywidgets-7.5.1-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/jupyterlab-1.2.4-pyhf63ae98_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/bokeh-1.4.0-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/h5py-2.9.0-py37h7918eee_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/matplotlib-3.1.1-py37h5429711_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/mkl_fft-1.0.15-py37ha843d7b_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/mkl_random-1.1.0-py37hd6b4f25_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/numpy-1.17.4-py37hc1035e2_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/holoviews-1.12.7-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/keras-applications-1.0.8-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/opt_einsum-3.1.0-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/pandas-0.25.3-py37he6710b0_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/scipy-1.3.2-py37h7c811a0_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/tensorboard-1.15.0-pyhb230dea_0.conda
+https://conda.anaconda.org/conda-forge/noarch/hyperopt-0.2.2-py_0.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/keras-preprocessing-1.1.0-py_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/linux-64/patsy-0.5.1-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/scikit-learn-0.22.1-py37hd81dba3_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/keras-base-2.2.4-py37_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/statsmodels-0.10.1-py37hdd07704_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/tensorflow-base-1.15.0-mkl_py37he1670d9_0.conda
+https://repo.anaconda.com/pkgs/main/noarch/seaborn-0.9.0-pyh91ea838_1.tar.bz2
+https://repo.anaconda.com/pkgs/main/noarch/tensorflow-estimator-1.15.1-pyh2649769_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/tensorflow-1.15.0-mkl_py37h28c19af_0.conda
+https://repo.anaconda.com/pkgs/main/linux-64/keras-2.2.4-0.conda

python_for_machine_learning_linux64_conda_specs.txt

@@ -0,0 +1,28 @@
+#| Source code
+
+This is source code that is either used in the presentation, or was developed
+to create it.  There is some material not covered in the presentation as well.
+
+## Requirements
+
+* Python version: at least 3.7
+* Packages (names listed that can be used with `pip` or `conda` to install):
+  * pandas
+  * xlrd
+  * seaborn
+  * matplotlib
+  * scipy
+  * jupyter
+  * scikit-learn
+  * keras
+  * hyperopt
+
+## What is it?
+* [`keras`](keras): illustration of using Keras for machine learning.
+* [`parameter-optimization`](parameter-optimization): example of parameter
+  optimization kusing hyperopt, although the examples do not optimize
+  hyperparameters in machine learning, that would be very similar.
+* [`scikit-learn`](scikit-learn): examples of scikit-learn for machine learning,
+  examples are provided forsupervised (regression and classification) and
+  unsupervised (clustering) learnign, as well as dimensionality reduction for
+  visualization of high-demensional data.

source-code/REEADME.md

@@ -0,0 +1,2 @@
+*.exe
+*.pbs.*

source-code/parameter-optimization/.gitignore

@@ -0,0 +1,12 @@
+CXX = g++
+CXXFLAGS = -fopenmp -std=c++14 -O2 -g -Wall -Wextra
+LDLIBS = -lm
+
+all: julia_omp.exe
+
+%.exe: %.o
+	$(CXX) $(CXXFLAGS) -o $@ $^ $(LDLIBS)
+
+clean:
+	$(RM) $(wildcard *.exe) $(wildcard *.o)
+	$(RM) core $(wildcard core.*)

source-code/parameter-optimization/JuliaJobs/Makefile

@@ -0,0 +1,17 @@
+# JuliaJobs
+
+Illustration of optimizing application parameters by running jobs.
+
+## What is it?
+
+  1. `julia_omp.cpp`: C++ application that compute the Julia set
+    fractal, using OpenMP for parallelization.  The parallel
+    for-loop schedule is `runtime`, and hence can be controled
+    using the `OMP_SCHEDULE` environment variable. Timing
+    information is writting to standard error.
+  1. `Makefile`: make file to build the Julia set application.
+  1. `julia.pbs`: PBS script to run the Julia set applciation
+    as a job.
+  1. `julia_optimize.py`: Python script to find optimal OpenMP
+    schedule, chunk size and number of threads for the Julia set
+    application.

source-code/parameter-optimization/JuliaJobs/README.md

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+#PBS -A lp_sys
+#PBS -l pmem=2gb
+#PBS -l walltime=00:05:00
+#PBS -j oe
+
+cd $PBS_O_WORKDIR
+
+source switch_to_2018a
+module load foss
+
+# verify that OpenMP environment variables have been passed
+# using -v options
+if [[ -z "$schedule" ]]
+then
+    (>&2 echo "### error: OpenMP schedule not set")
+    exit 1
+fi
+if [[ -z "$chunk" ]]
+then
+    (>&2 echo "### error: OpenMP chunk not set")
+    exit 1
+fi
+
+OMP_SCHEDULE="${schedule},${chunk}"
+(>&2 echo "### info: OMP_SCHEDULE=\'${OMP_SCHEDULE}\'")
+
+if [[ -z "$OMP_NUM_THREADS" ]]
+then
+    (>&2 echo "### error: OpenMP number of threads not set")
+    exit 2
+else
+    (>&2 echo "### info: OMP_NUM_THREADS=\'${OMP_NUM_THREADS}\'")
+fi
+
+# ensure the executable is op to date
+make all
+if [[ $? -ne 0 ]]
+then
+    (>&2 echo "### error: make failed")
+    exit 3
+fi
+
+# determine time file name
+time_file="julia.pbs.time${PBS_JOBID/\.*/}"
+
+# run application on a sufficiently large matrix, disregarding
+# output, timing information is writting to a file
+for i in $(seq 3)
+do
+    ./julia_omp.exe 4096 > /dev/null 2>> "${time_file}"
+done