|
2 | 2 |
|
3 | 3 | Kannon is a wrapper for the [gokart](https://github.com/m3dev/gokart) library that allows gokart tasks to be easily executed in a distributed and parallel manner on multiple [kubernetes](https://kubernetes.io/) jobs. |
4 | 4 |
|
| 5 | +# Install |
| 6 | +Kannon can be installed via `pip`. |
| 7 | + |
| 8 | +```bash |
| 9 | +pip install kannon |
| 10 | +``` |
| 11 | + |
| 12 | +# Usage |
| 13 | +It is required for users to prepare following two scripts and copy them into a docker container: |
| 14 | +- A script to start task pipeline on master job. |
| 15 | +- A script for child jobs to run assigned tasks. |
| 16 | + |
| 17 | +Easy and self-contained quick starter will be available soon! |
| 18 | + |
| 19 | +## A script to run job on master job |
| 20 | +Required to: |
| 21 | +- Import module where `gokart.TaskOnKart` and `kannon.TaskOnBullet` classes are defined. |
| 22 | +- Load luigi and k8s configs. |
| 23 | +- Create a task instance. |
| 24 | +- Run `Kannon.build`. |
| 25 | + |
| 26 | +```python |
| 27 | +""" This script runs on master job. """ |
| 28 | +import logging |
| 29 | + |
| 30 | +import gokart |
| 31 | +import luigi |
| 32 | +from kubernetes import config, client |
| 33 | +import fire |
| 34 | + |
| 35 | +# TODO: Import task definition here! |
| 36 | +import example_tasks |
| 37 | +from kannon import Kannon |
| 38 | + |
| 39 | +logging.basicConfig(level=logging.INFO) |
| 40 | + |
| 41 | + |
| 42 | +def main( |
| 43 | + container_name: str, |
| 44 | + image_name: str, |
| 45 | +): |
| 46 | + # TODO: Load luigi config here! |
| 47 | + luigi.configuration.LuigiConfigParser.add_config_path("./conf/base.ini") |
| 48 | + |
| 49 | + # TODO: Load kube config here! |
| 50 | + config.load_incluster_config() |
| 51 | + v1 = client.BatchV1Api() |
| 52 | + # TODO: Create task instance here! |
| 53 | + task_root = [CREATE TASK INSTANCE HERE] |
| 54 | + |
| 55 | + # TODO: Run Kannon.build! |
| 56 | + Kannon( |
| 57 | + api_instance=v1, |
| 58 | + namespace="mynamespace", |
| 59 | + image_name=image_name, |
| 60 | + container_name=container_name, |
| 61 | + job_prefix="myjob", |
| 62 | + path_child_script="./run_child.py", |
| 63 | + env_to_inherit=["TASK_WORKSPACE_DIRECTORY"], |
| 64 | + service_account_name="myserviceaccount, |
| 65 | + ).build(task_root) |
| 66 | + |
| 67 | + |
| 68 | +if __name__ == "__main__": |
| 69 | + Fire.fire(main) |
| 70 | +``` |
| 71 | + |
| 72 | +## A script for child jobs to run assigned tasks |
| 73 | +For now, it is required for users to prepare the following script. In the future release, it will not be required. |
| 74 | + |
| 75 | +Required to: |
| 76 | +- Import module where `gokart.TaskOnKart` and `kannon.TaskOnBullet` classes are defined. |
| 77 | +- Load luigi config. |
| 78 | +- Parse a serialized task instance. |
| 79 | +- Run `gokart.build`. |
| 80 | + |
| 81 | +```python |
| 82 | +""" This script requires to be defined by user. """ |
| 83 | +import gokart |
| 84 | +import luigi |
| 85 | +import logging |
| 86 | + |
| 87 | +# TODO: Import task definitions here! |
| 88 | +import example_tasks |
| 89 | + |
| 90 | +import fire |
| 91 | + |
| 92 | +logging.basicConfig(level=logging.INFO) |
| 93 | + |
| 94 | + |
| 95 | +def main(serialized_task: str): |
| 96 | + # TODO: Load luigi config here! |
| 97 | + luigi.configuration.LuigiConfigParser.add_config_path("./conf/base.ini") |
| 98 | + |
| 99 | + # TODO: Parse a serialized gokart.TaskOnKart here! |
| 100 | + task: gokart.TaskOnKart = gokart.TaskInstanceParameter().parse(serialized_task) |
| 101 | + # TODO: Run gokart.build! |
| 102 | + gokart.build(task) |
| 103 | + |
| 104 | + |
| 105 | +if __name__ == "__main__": |
| 106 | + fire.Fire(main) |
| 107 | +``` |
| 108 | + |
5 | 109 | # Thanks |
6 | 110 |
|
7 | 111 | Kannon is a wrapper for gokart. Thanks to gokart and dependent projects! |
|
0 commit comments