Updates for latest Faasm (#12)

* Update to removal of shared_store dir

* Adding remote upload for funds

* Add invoke and upload functions

* Add logging for local function upload

* Fix printing error

Author: Shillaker

README.md

@@ -1,4 +1,4 @@
-# Faasm Python Environment [![Tests](https://github.com/faasm/python/workflows/Tests/badge.svg?branch=master)](https://github.com/faasm/python/actions)  [![License](https://img.shields.io/github/license/faasm/python.svg)](https://github.com/faasm/python/blob/master/LICENSE.md) 
+# Faasm Python Environment [![Tests](https://github.com/faasm/python/workflows/Tests/badge.svg?branch=master)](https://github.com/faasm/python/actions)  [![License](https://img.shields.io/github/license/faasm/python.svg)](https://github.com/faasm/python/blob/master/LICENSE.md)
 
 This build cross-compiles CPython and a number of Python modules to WebAssembly
 for use in [Faasm](https://github.com/faasm/faasm).
@@ -10,9 +10,9 @@ interface](https://github.com/faasm/faasm/blob/master/docs/host_interface.md).
 ## Set-up and development
 
 This repo is developed using the Faasm development environment, set up according
-to [the docs](https://github.com/faasm/faasm/blob/master/docs/development.md). 
+to [the docs](https://github.com/faasm/faasm/blob/master/docs/development.md).
 
-To set up your local environment, run the `python` CLI as per the Faasm docs, 
+To set up your local environment, run the `python` CLI as per the Faasm docs,
 then:
 
 ```bash
@@ -42,7 +42,7 @@ inv runtime
 inv func
 
 # Copy the actual Python functions into place
-inv func.uploadpy --local
+inv func.upload-all --local
 ```
 
 ## Code changes
@@ -56,10 +56,10 @@ compare](https://github.com/python/cpython/compare/v3.8.2...faasm:faasm).
 A similar (small) list of changes for numpy can be seen
 [here](https://github.com/numpy/numpy/compare/v1.19.2...faasm:faasm).
 
-CPython is built statically, some notes on this process can be found 
-[here](https://wiki.python.org/moin/BuildStatically). 
+CPython is built statically, some notes on this process can be found
+[here](https://wiki.python.org/moin/BuildStatically).
 
-Several of the code changes to CPython and numpy were borrowed from 
+Several of the code changes to CPython and numpy were borrowed from
 [pyodide](https://github.com/iodide-project/pyodide).
 
 ## Releasing
@@ -107,7 +107,7 @@ scripts on the build machine (not any other Python that might be installed).
 
 Do **not** upgrade Pip in the build machine copy of Python.
 
-The versions of Pip and Python for wasm and the build machine must match 
+The versions of Pip and Python for wasm and the build machine must match
 exactly.
 
 ### Building CPython to WebAssembly
@@ -121,10 +121,10 @@ inv cpython
 The result is installed at `third-party/cpython/install/wasm`.
 
 We provide a [Setup.local](third-party/cpython/Modules/Setup.local) file, which
-specifies which standard CPython modules will be built statically. 
+specifies which standard CPython modules will be built statically.
 
 At the end of the CPython build, it will print out a list of which modules have
-been successfully built and which have failed. Note that many of the standard 
+been successfully built and which have failed. Note that many of the standard
 library modules will fail in this build, but the ones we need should succeed.
 
 ## Cross-compilation set-up
@@ -137,13 +137,13 @@ See the dev instructions above for set-up.
 
 ### Changing the crossenv environment
 
-Crossenv picks up the cross-compilation environment from the CPython 
-build artifacts. Therefore, to make changes to the cross-compilation 
+Crossenv picks up the cross-compilation environment from the CPython
+build artifacts. Therefore, to make changes to the cross-compilation
 environment:
 
 - Modify the CPython build (see `tasks.py`)
-- Rerun the CPython build (`inv cpython --clean`) 
-- Rebuild the crossenv (`./bin/crossenv_setup.sh`) 
+- Rerun the CPython build (`inv cpython --clean`)
+- Rebuild the crossenv (`./bin/crossenv_setup.sh`)
 - Enter the crossenv and inspect the environment with `bin/sanity_check.py`
 
 ## Modules
@@ -155,7 +155,7 @@ To run this you must first have the cross-env activated as described above.
 
 ```bash
 # Install all supported modules
-inv libs.install 
+inv libs.install
 
 # Install experimental modules
 inv libs.install --experimental
@@ -167,7 +167,7 @@ inv libs.install --name numpy
 inv libs.install --name <module_name>
 ```
 
-Libraries will then be installed to 
+Libraries will then be installed to
 `cross_venv/cross/lib/python3.8/site-packages`.
 
 ### Debugging module builds
@@ -206,7 +206,7 @@ inv libs.install --name mxnet
 inv libs.install --name horovod
 ```
 
-### Cleaning and uninstalling 
+### Cleaning and uninstalling
 
 ```
 # Clean then rebuild

tasks/func.py

@@ -1,8 +1,10 @@
 from faasmtools.compile_util import wasm_cmake
 from faasmtools.env import WASM_DIR
+from faasmtools.build import FAASM_LOCAL_DIR
+
+import requests
 from os.path import join
 from invoke import task
-from faasmtools.build import FAASM_LOCAL_DIR
 from os import makedirs, listdir
 from shutil import copy
 from tasks.env import PROJ_ROOT
@@ -17,7 +19,7 @@
 DEST_FOLDER = join(WASM_DIR, USER, FUNC)
 DEST_FILE = join(DEST_FOLDER, "function.wasm")
 
-FAASM_SHARED_STORAGE_ROOT = join(FAASM_LOCAL_DIR, "shared_store")
+FAASM_SHARED_STORAGE_ROOT = join(FAASM_LOCAL_DIR, "shared")
 PY_FUNC_DIR = join(PROJ_ROOT, "func", "python")
 PY_UPLOAD_DIR = join(FAASM_SHARED_STORAGE_ROOT, "pyfuncs", "python")
 
@@ -31,30 +33,76 @@ def compile(ctx, clean=False, debug=False):
     wasm_cmake(FUNC_DIR, FUNC_BUILD_DIR, FUNC, clean, debug)
 
     # Copy the wasm into place
-
     makedirs(DEST_FOLDER, exist_ok=True)
     copy(WASM_FILE, DEST_FILE)
 
 
 @task
-def uploadpy(ctx, local=False):
+def upload(ctx, host="upload", port=8002):
     """
-    Upload functions written in Python
+    Upload the CPython function
     """
-    if not local:
-        raise RuntimeError("Remote upload not yet implemented")
+    url = "http://{}:{}/f/{}/{}".format(host, port, USER, FUNC)
+    print("Uploading {}/{} to {}".format(USER, FUNC, url))
+    response = requests.put(url, data=open(WASM_FILE, "rb"))
 
-    makedirs(PY_UPLOAD_DIR, exist_ok=True)
+    print("Response {}: {}".format(response.status_code, response.text))
 
-    # Get all Python funcs
-    funcs = listdir(PY_FUNC_DIR)
-    funcs = [f for f in funcs if f.endswith(".py")]
-    funcs = [f.replace(".py", "") for f in funcs]
 
-    for func in funcs:
+@task
+def uploadpy(ctx, func, local=False, host="upload", port=8002):
+    src_file = join(PY_FUNC_DIR, "{}.py".format(func))
+
+    if local:
+        makedirs(PY_UPLOAD_DIR, exist_ok=True)
         func_upload_dir = join(PY_UPLOAD_DIR, func)
         makedirs(func_upload_dir, exist_ok=True)
 
-        src_file = join(PY_FUNC_DIR, "{}.py".format(func))
         dest_file = join(func_upload_dir, "function.py")
+        print("Copying function {} {} -> {}".format(func, src_file, dest_file))
         copy(src_file, dest_file)
+    else:
+        url = "http://{}:{}/p/{}/{}".format(host, port, "python", func)
+        response = requests.put(url, data=open(src_file, "rb"))
+
+        print(
+            "{} response {}: {}".format(
+                func, response.status_code, response.text
+            )
+        )
+
+
+@task
+def upload_all(ctx, local=False, host="upload", port=8002):
+    """
+    Upload all Python functions
+    """
+    funcs = listdir(PY_FUNC_DIR)
+    funcs = [f for f in funcs if f.endswith(".py")]
+    funcs = [f.replace(".py", "") for f in funcs]
+
+    for func in funcs:
+        uploadpy(ctx, func, local=local, host=host, port=port)
+
+
+@task
+def invoke(ctx, user, func, input_data=None, host="worker", port=8080):
+    """
+    Invoke a function
+    """
+
+    url = "http://{}:{}".format(host, port)
+    data = {
+        "user": USER,
+        "function": FUNC,
+        "py_user": user,
+        "py_func": func,
+        "python": True,
+    }
+
+    if input_data:
+        data["input_data"] = input_data
+
+    response = requests.post(url, json=data)
+
+    print("Response {}:\n{}".format(response.status_code, response.text))