Merge pull request #224 from JosepSampe/pywren-dev

New docker_executor()

Author: gilv

CHANGELOG.md

@@ -4,6 +4,10 @@
 
 ### Added
 - New local_executor() to run pywren jobs in the local machine
+- New localhost compute backend
+- New localhost storage backend
+- New docker_executor() to run pywren jobs in the local machine by using docker
+- New docker compute backend
 
 ### Changed
 - Docs updated

docs/api-details.md

@@ -1,17 +1,18 @@
 # PyWren API Details
 
 ## Executor
-The primary object in PyWren is the executor. The standard way to get everything set up is to import pywren_ibm_cloud, and call on of the available methods to get a ready-to-use executor. The available executors are: `ibm_cf_executor()`, `knative_executor()` and `function_executor()`
+The primary object in PyWren is the executor. The standard way to get everything set up is to import pywren_ibm_cloud, and call on of the available methods to get a ready-to-use executor. The available executors are: `ibm_cf_executor()`, `knative_executor()`, `function_executor()`, `local_executor()` and `docker_executor()`
 
-**ibm_cf_executor(\*\*kwargs)**, **knative_executor(\*\*kwargs)**, **function_executor(\*\*kwargs)**
+**ibm_cf_executor(\*\*kwargs)**
 
-Initialize and return an executor object. All the parameters set in the executor will overwrite those set in the configuration.
+Initialize and return an IBM Cloud Functions executor object. All the parameters set in the executor will overwrite those set in the configuration.
 
 |Parameter | Default | Description|
 |---|---|---|
 |config | None | Settings passed in here will override those in pywren_config|
 |runtime |  None | Name of the docker image to run the functions |
 |runtime_memory | 256 | Memory (in MB) to use to run the functions |
+|storage_backend | ibm_cos | Storage backend to store temp data|
 |rabbitmq_monitor | False | Activate RabbitMQ monitoring |
 |log_level | None | Log level printing (INFO, DEBUG, ...) |
 
@@ -21,6 +22,82 @@ import pywren_ibm_cloud as pywren
 pw = pywren.ibm_cf_executor()
 ```
 
+**knative_executor(\*\*kwargs)**
+
+Initialize and return a Knative executor object. All the parameters set in the executor will overwrite those set in the configuration.
+
+|Parameter | Default | Description|
+|---|---|---|
+|config | None | Settings passed in here will override those in pywren_config|
+|runtime |  None | Name of the docker image to run the functions |
+|runtime_memory | 256 | Memory (in MB) to use to run the functions |
+|storage_backend | ibm_cos | Storage backend to store temp data|
+|rabbitmq_monitor | False | Activate RabbitMQ monitoring |
+|log_level | None | Log level printing (INFO, DEBUG, ...) |
+
+Usage:
+```python
+import pywren_ibm_cloud as pywren
+pw = pywren.knative_executor()
+```
+
+**function_executor(\*\*kwargs)**
+
+Initialize and return a generic executor object. All the parameters set in the executor will overwrite those set in the configuration.
+
+|Parameter | Default | Description|
+|---|---|---|
+|config | None | Settings passed in here will override those in pywren_config|
+|runtime |  None | Name of the docker image to run the functions |
+|runtime_memory | 256 | Memory (in MB) to use to run the functions |
+|backend | ibm_cf | name of the compute backend to run the functions |
+|storage_backend | ibm_cos | Storage backend to store temp data|
+|rabbitmq_monitor | False | Activate RabbitMQ monitoring |
+|log_level | None | Log level printing (INFO, DEBUG, ...) |
+
+Usage:
+```python
+import pywren_ibm_cloud as pywren
+pw = pywren.function_executor()
+```
+
+**local_executor(\*\*kwargs)**
+
+Initialize and return a Localhost executor object. This executor runs the functions in local processes. All the parameters set in the executor will overwrite those set in the configuration.
+
+
+|Parameter | Default | Description|
+|---|---|---|
+|config | None | Settings passed in here will override those in pywren_config|
+|storage_backend | localhost | Storage backend to store temp data |
+|rabbitmq_monitor | False | Activate RabbitMQ monitoring |
+|log_level | None | Log level printing (INFO, DEBUG, ...) |
+
+Usage:
+```python
+import pywren_ibm_cloud as pywren
+pw = pywren.local_executor()
+```
+
+**docker_executor(\*\*kwargs)**
+
+Initialize and return a Docker executor object. This executor runs the functions in local Dockers. All the parameters set in the executor will overwrite those set in the configuration.
+
+
+|Parameter | Default | Description|
+|---|---|---|
+|config | None | Settings passed in here will override those in pywren_config|
+|runtime |  None | Name of the docker image to run the functions |
+|storage_backend | localhost | Storage backend to store temp data |
+|rabbitmq_monitor | False | Activate RabbitMQ monitoring |
+|log_level | None | Log level printing (INFO, DEBUG, ...) |
+
+Usage:
+```python
+import pywren_ibm_cloud as pywren
+pw = pywren.docker_executor()
+```
+
 ## Executor.call_async()
 Spawn only one function activation.
 

pywren_ibm_cloud/__init__.py

@@ -22,7 +22,7 @@
 
 
 def ibm_cf_executor(config=None, runtime=None, runtime_memory=None,
-                    compute_backend_region=None, storage_backend=None,
+                    region=None, storage_backend=None,
                     storage_backend_region=None, rabbitmq_monitor=None,
                     log_level=None):
     """
@@ -32,26 +32,25 @@ def ibm_cf_executor(config=None, runtime=None, runtime_memory=None,
     return FunctionExecutor(
         config=config, runtime=runtime, runtime_memory=runtime_memory,
         compute_backend=compute_backend,
-        compute_backend_region=compute_backend_region,
+        compute_backend_region=region,
         storage_backend=storage_backend,
         storage_backend_region=storage_backend_region,
         rabbitmq_monitor=rabbitmq_monitor,
         log_level=log_level
     )
 
 
-def knative_executor(config=None, runtime=None, runtime_memory=None,
-                     compute_backend_region=None, storage_backend=None,
-                     storage_backend_region=None, rabbitmq_monitor=None,
-                     log_level=None):
+def knative_executor(config=None, runtime=None, runtime_memory=None, region=None,
+                     storage_backend=None, storage_backend_region=None,
+                     rabbitmq_monitor=None, log_level=None):
     """
     Function executor for Knative
     """
     compute_backend = 'knative'
     return FunctionExecutor(
         config=config, runtime=runtime, runtime_memory=runtime_memory,
         compute_backend=compute_backend,
-        compute_backend_region=compute_backend_region,
+        compute_backend_region=region,
         storage_backend=storage_backend,
         storage_backend_region=storage_backend_region,
         rabbitmq_monitor=rabbitmq_monitor,
@@ -60,16 +59,17 @@ def knative_executor(config=None, runtime=None, runtime_memory=None,
 
 
 def function_executor(config=None, runtime=None, runtime_memory=None,
-                      compute_backend=None, compute_backend_region=None,
+                      backend=None, region=None,
                       storage_backend=None, storage_backend_region=None,
                       rabbitmq_monitor=None, log_level=None):
     """
     Generic function executor
     """
     return FunctionExecutor(
-        config=config, runtime=runtime, runtime_memory=runtime_memory,
-        compute_backend=compute_backend,
-        compute_backend_region=compute_backend_region,
+        config=config, runtime=runtime,
+        runtime_memory=runtime_memory,
+        compute_backend=backend,
+        compute_backend_region=region,
         storage_backend=storage_backend,
         storage_backend_region=storage_backend_region,
         rabbitmq_monitor=rabbitmq_monitor,
@@ -94,3 +94,24 @@ def local_executor(config=None, storage_backend=None, storage_backend_region=Non
         rabbitmq_monitor=rabbitmq_monitor,
         log_level=log_level
     )
+
+
+def docker_executor(config=None, runtime=None, storage_backend=None,
+                    storage_backend_region=None, rabbitmq_monitor=None,
+                    log_level=None):
+    """
+    Localhost function executor
+    """
+    compute_backend = 'docker'
+
+    if storage_backend is None:
+        storage_backend = 'localhost'
+
+    return FunctionExecutor(
+        config=config, runtime=runtime,
+        compute_backend=compute_backend,
+        storage_backend=storage_backend,
+        storage_backend_region=storage_backend_region,
+        rabbitmq_monitor=rabbitmq_monitor,
+        log_level=log_level
+    )

pywren_ibm_cloud/compute/backends/docker/__init__.py

@@ -0,0 +1 @@
+from .docker import DockerBackend as ComputeBackend

pywren_ibm_cloud/compute/backends/docker/config.py

@@ -0,0 +1,68 @@
+import sys
+import multiprocessing
+from pywren_ibm_cloud.utils import version_str
+
+RUNTIME_DEFAULT_35 = 'pywren-docker-runtime-v3.5:latest'
+RUNTIME_DEFAULT_36 = 'pywren-docker-runtime-v3.6:latest'
+RUNTIME_DEFAULT_37 = 'pywren-docker-runtime-v3.7:latest'
+RUNTIME_TIMEOUT_DEFAULT = 600  # 10 minutes
+RUNTIME_MEMORY_DEFAULT = 256  # 256 MB
+
+_DOCKERFILE_DEFAULT = """
+RUN apt-get update && apt-get install -y \
+        git
+
+RUN pip install --upgrade pip setuptools six \
+    && pip install --no-cache-dir \
+        pika==0.13.1 \
+        ibm-cos-sdk \
+        redis \
+        requests \
+        numpy
+
+# Copy PyWren app to the container image.
+ENV APP_HOME /pywren
+WORKDIR $APP_HOME
+
+RUN git clone https://github.com/pywren/pywren-ibm-cloud && cd pywren-ibm-cloud && pip install .
+
+# entry_point.py is automatically generated. Do not modify next lines!
+COPY entry_point.py .
+
+ENTRYPOINT ["python", "entry_point.py"]
+CMD []
+"""
+
+DOCKERFILE_DEFAULT_35 = """
+FROM python:3.5-slim-buster
+""" + _DOCKERFILE_DEFAULT
+
+DOCKERFILE_DEFAULT_36 = """
+FROM python:3.6-slim-buster
+""" + _DOCKERFILE_DEFAULT
+
+DOCKERFILE_DEFAULT_37 = """
+FROM python:3.7-slim-buster
+""" + _DOCKERFILE_DEFAULT
+
+
+def load_config(config_data):
+    if 'runtime_memory' not in config_data['pywren']:
+        config_data['pywren']['runtime_memory'] = RUNTIME_MEMORY_DEFAULT
+    if 'runtime_timeout' not in config_data['pywren']:
+        config_data['pywren']['runtime_timeout'] = RUNTIME_TIMEOUT_DEFAULT
+    if 'runtime' not in config_data['pywren']:
+        this_version_str = version_str(sys.version_info)
+        if this_version_str == '3.5':
+            config_data['pywren']['runtime'] = RUNTIME_DEFAULT_35
+        elif this_version_str == '3.6':
+            config_data['pywren']['runtime'] = RUNTIME_DEFAULT_36
+        elif this_version_str == '3.7':
+            config_data['pywren']['runtime'] = RUNTIME_DEFAULT_37
+
+    total_cores = multiprocessing.cpu_count()
+    if 'docker' not in config_data:
+        config_data['docker'] = {}
+        config_data['docker']['workers'] = total_cores
+    elif 'workers' not in config_data['docker']:
+        config_data['docker']['workers'] = total_cores