Skip to content

Commit eef156e

Browse files
authored
Merge pull request #3 from m3dev/doc/readme
[Doc] Add install and usage sections to README
2 parents d1da9e2 + 19ab7ab commit eef156e

File tree

1 file changed

+104
-0
lines changed

1 file changed

+104
-0
lines changed

README.md

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,110 @@
22

33
Kannon is a wrapper for the [gokart](https://github.com/m3dev/gokart) library that allows gokart tasks to be easily executed in a distributed and parallel manner on multiple [kubernetes](https://kubernetes.io/) jobs.
44

5+
# Install
6+
Kannon can be installed via `pip`.
7+
8+
```bash
9+
pip install kannon
10+
```
11+
12+
# Usage
13+
It is required for users to prepare following two scripts and copy them into a docker container:
14+
- A script to start task pipeline on master job.
15+
- A script for child jobs to run assigned tasks.
16+
17+
Easy and self-contained quick starter will be available soon!
18+
19+
## A script to run job on master job
20+
Required to:
21+
- Import module where `gokart.TaskOnKart` and `kannon.TaskOnBullet` classes are defined.
22+
- Load luigi and k8s configs.
23+
- Create a task instance.
24+
- Run `Kannon.build`.
25+
26+
```python
27+
""" This script runs on master job. """
28+
import logging
29+
30+
import gokart
31+
import luigi
32+
from kubernetes import config, client
33+
import fire
34+
35+
# TODO: Import task definition here!
36+
import example_tasks
37+
from kannon import Kannon
38+
39+
logging.basicConfig(level=logging.INFO)
40+
41+
42+
def main(
43+
container_name: str,
44+
image_name: str,
45+
):
46+
# TODO: Load luigi config here!
47+
luigi.configuration.LuigiConfigParser.add_config_path("./conf/base.ini")
48+
49+
# TODO: Load kube config here!
50+
config.load_incluster_config()
51+
v1 = client.BatchV1Api()
52+
# TODO: Create task instance here!
53+
task_root = [CREATE TASK INSTANCE HERE]
54+
55+
# TODO: Run Kannon.build!
56+
Kannon(
57+
api_instance=v1,
58+
namespace="mynamespace",
59+
image_name=image_name,
60+
container_name=container_name,
61+
job_prefix="myjob",
62+
path_child_script="./run_child.py",
63+
env_to_inherit=["TASK_WORKSPACE_DIRECTORY"],
64+
service_account_name="myserviceaccount,
65+
).build(task_root)
66+
67+
68+
if __name__ == "__main__":
69+
Fire.fire(main)
70+
```
71+
72+
## A script for child jobs to run assigned tasks
73+
For now, it is required for users to prepare the following script. In the future release, it will not be required.
74+
75+
Required to:
76+
- Import module where `gokart.TaskOnKart` and `kannon.TaskOnBullet` classes are defined.
77+
- Load luigi config.
78+
- Parse a serialized task instance.
79+
- Run `gokart.build`.
80+
81+
```python
82+
""" This script requires to be defined by user. """
83+
import gokart
84+
import luigi
85+
import logging
86+
87+
# TODO: Import task definitions here!
88+
import example_tasks
89+
90+
import fire
91+
92+
logging.basicConfig(level=logging.INFO)
93+
94+
95+
def main(serialized_task: str):
96+
# TODO: Load luigi config here!
97+
luigi.configuration.LuigiConfigParser.add_config_path("./conf/base.ini")
98+
99+
# TODO: Parse a serialized gokart.TaskOnKart here!
100+
task: gokart.TaskOnKart = gokart.TaskInstanceParameter().parse(serialized_task)
101+
# TODO: Run gokart.build!
102+
gokart.build(task)
103+
104+
105+
if __name__ == "__main__":
106+
fire.Fire(main)
107+
```
108+
5109
# Thanks
6110

7111
Kannon is a wrapper for gokart. Thanks to gokart and dependent projects!

0 commit comments

Comments
 (0)