Merge pull request #12 from CodersOfTheNight/augment_config_and_doc

Augment config and doc

Author: zaibacu

.travis.yml

@@ -1,6 +1,7 @@
 language: python
 python:
 - 3.5
+- 3.6
 
 install:
   - pip install -r requirements/release.txt

AUTHORS.md

@@ -0,0 +1,3 @@
+Maintainers
+============
+Šarūnas Navickas (zaibacu@gmail.com): 2016 - now

CONTRIBUTING.md

@@ -0,0 +1,21 @@
+New agents
+==========
+Most natural way of contributing to this project is creating new agents.
+What is agent? It is basically a metrics collector for specific service, 
+eg. `oshino-redis` is an agent which runs `info` command as a Redis client,
+parses received metrics and gives them to `oshino` as events.
+
+To make life easier, there's a `cookiecutter` template [oshino-cookiecutter](https://github.com/CodersOfTheNight/oshino-cookiecutter)
+which generates all boilerplate code, you just need to modify it.
+
+Already existing custom agents are mostly listed here: [Third party Agents](https://github.com/CodersOfTheNight/oshino/blob/master/docs/thirdparty.md)
+
+Contributing to the Core
+========================
+It should mostly consist of reporting/solving issues. 
+
+General Guidelines
+===================
+- Code must pass `flake8` (PEP8) coding style test
+- All code (with few exceptions) must be non-blocking - asyncio compatible
+- New parameters should be covered in documentation section

README.md

@@ -26,6 +26,11 @@ How to install
 ==============
 `pip install oshino`
 
+Quickstart
+==========
+It is highly recommended for new users to use [Quickstart Guide](quickstart.md)
+
+
 Riemann. What? Why? How?
 =========================
 Riemann is a backbone of this system. It does alerting, it receives metrics, it aggregates metrics and it decides where to send them (eg.: Graphite, Logstash).
@@ -56,3 +61,7 @@ Documentation about additional agents can be found [here](docs/thirdparty.md)
 More documentation
 ==================
 More documentation can be found under [docs](docs/index.md) directory
+
+Contributing
+============
+Refer to [CONTRIBUTING.md](CONTRIBUTING.md)

docs/agents.md

@@ -0,0 +1,44 @@
+What is "Agent"?
+-----------------
+In this context, we call a worker which collects specific type of metrics, an agent.
+
+How to use them
+--------------
+To be able to use any of agents, first you must import them.
+Importing is done via config, by adding them under `agents` section
+
+
+An example config could look like this:
+```yaml
+---
+interval: 10
+loglevel: DEBUG
+riemann:
+  host: localhost
+  port: 5555
+  transport: BlankTransport
+agents:
+  - name: echo
+    module: oshino.agents.subprocess_agent.SubprocessAgent
+    script: "echo 'Hello world!'"
+    tag: "bash"
+```
+
+More configuration info can be found under [Config](config.md)
+
+As you can see, there's one agent in agents array, which is called `echo`
+and uses internal `SubprocessAgent` class. 
+This type of agent is able to execute command in command line
+and return it's result as a metric.
+
+Internal agents also includes `HttpAgent` which is able to do HTTP calls,
+and returns response and time it took to execute as metrics.
+
+Usual source of agents is [Third Party agent section](thirdparty.md)
+They can be installed via `pip` command, or using `oshino-admin`. 
+For more information on `oshino-admin` usage, do `oshino-admin --help`.
+
+
+Creating custom Agent
+---------------------
+It can be done by using our [Cookiecutter Template](https://github.com/CodersOfTheNight/oshino-cookiecutter)

docs/augments.md

@@ -0,0 +1,27 @@
+What is "Augment"?
+------------------
+In this context, we a process which reads specific metrics and produces new ones, we call an augment.
+Augment's job is to produce additional insights from metrics we already have.
+
+How to use them?
+----------------
+You need to include specific augment inside config under augments array, very similar to [Agents](agents.md).
+
+An example config could look like this:
+```yaml
+---
+interval: 10
+loglevel: DEBUG
+riemann:
+  host: localhost
+  port: 5555
+  transport: BlankTransport
+augments:
+  - name: moving average
+    key: cpu 
+    module: oshino.augments.stats.SimpleMovingAverage
+    step: 5
+    tag: "sma"
+```
+
+More configuration info can be found under [Config](config.md)

docs/config.md

@@ -1,33 +1,54 @@
 Base Config
------------
+============
 - `interval` - how often to send metrics (in seconds)
 - `riemann.host` - riemann hostname (default: localhost)
 - `riemann.port` - riemann port (default: 5555)
 - `riemann.transport` - set transport protocol for riemann (default: TCPTransport)
 - `loglevel` - level of logging (default: INFO)
 - `sentry-dsn` - sentry address for error reporting (default: `None`)
+- `executor` - allows to define executor class, default one is `concurrent.futures.ThreadPoolExecutor`
+- `executors-count` - set worker count for executor, default: 10
+- `agents` - an array of agent configs used in this setup
+- `augments` - an array of augment configs used in this setup
+
+Agents
+=======
 
 General Config for Agents
 ------------------------
+- `name`- give name for agent
 - `tag` - adds this tag to all metrics related to this agent
+- `module` - path to module from which actual agent should be loaded. In most cases this path should be declared in actor's README
 
 Http Agent Config
 ----------------
+Agent used to produce HTTP calls to specific website
+module: `oshino.agents.http_agent.HttpAgent`
+
 - `url` - which url to call
 
 Subprocess Agent Config
 -----------------------
+Agent used to execute CLI scripts
+module: `oshino.agents.subprocess_agent.SubprocessAgent`
+
 - `script` - what shell script to execute
 
 
-Suggested Riemann config
--------------------------
-```clojure
- (where (tagged-all ["oshino", "heartbeat"])
-       (changed :state
-         (email "something@something.com")
-       )
-    )
-```
+Augments
+========
+
+General config for Augments
+---------------------------
+- `name`- give name for augment
+- `key` - defines a metric key to listen to
+- `tag` - adds this tag to all metrics related to this augment
+- `module` - path to module from which actual augment should be loaded. In most cases this path should be declared in augment's README
+
+
+Moving Average
+--------------
+Reads metrics and produces moving average
+module: `oshino.augments.stats.MovingAverage`
 
-If Oshino goes down, or encounters some performance hickups - you'll be notified
+- `step` - how much metrics it should inhale before it can produce average

docs/index.md

@@ -1,9 +1,12 @@
 Table of Contents:
 ==================
+- [Quickstart](quickstart.md)
 - [Installiation](install.md)
 - [Riemann](riemann.md)
 - Usage
     - [Events](events.md)
     - [Config](config.md)
 - Extending
+    - [Agents](agents.md)
+    - [Augments](augments.md)
     - [Third Party Agents](thirdparty.md)

docs/quickstart.md

@@ -0,0 +1,106 @@
+Preparing Environment
+======================
+You will need Python version 3.5+ and `pip` installed.
+In the most cases `pip` should come bundled, if building from source, you will need to have `openssl` development package installed, otherwise it will skip.
+
+If for some reason you don't have it and cannot build with `openssl`, there's always:
+
+`wget https://bootstrap.pypa.io/get-pip.py | python`
+
+Virtualenv
+----------
+It is optional, but highly recommended. 
+Can be installed via `pip install virtualenv`
+
+To create actual virtualenv, do:
+`virtualenv venv`
+Or if you have multiple versions of python (usually 2.7 and 3.****), 
+failproof approach would be:
+`python3 -m virtualenv venv`
+
+This will create virtual environment inside folder
+
+Depending on OS, activating environment may slightly differ.
+Linux/MacOS/*BSD it is `source venv/bin/activate`
+Windows: `venv\Scripts\activate`
+
+Installing Oshino
+-----------------
+When everything is prepared, installing is as easy as:
+`pip install oshino`
+
+
+Launching Oshino
+================
+First create config. Easy way to do that is by using `oshino-admin`
+
+`oshino-admin config init config.yaml`
+
+And now you can actually start it:
+`oshino --config=config.yaml`
+
+Running as daemon
+------------------
+To start as a daemon, we need to use `oshino-admin`
+`oshino-admin start --config=config.yaml`
+
+If you're running on your local machine or not as root, it's possible to get error like:
+`Unable to create the pidfile.`
+By default it writes PID (Process ID) file to `/var/run/oshino.pid` and permissions are not always sufficient to do that.
+Issue can be easily mitigated by providing custom path:
+`oshino-admin start --config=config.yaml --pid=oshino.pid`
+
+To check if it's running:
+`oshino-admin status`
+
+To stop it:
+`oshino-admin stop`
+
+Main caveat in using custom pid path, you need to provide it for `status` and `stop` commands as well:
+
+`oshino-admin status --pid=oshino.pid`
+`oshino-admin stop --pid=oshino.pid`
+
+
+Installing plugins
+==================
+
+In the most cases, these plugins are for agents. Agent in Oshino context is metrics collector.
+So for example, if our service is using [Prometheus](https://prometheus.io/) and we want to collect metrics from it, 
+plugin needs to be installed:
+
+`oshino-admin plugin install oshino_prometheus`
+
+To list available plugins:
+`oshino-admin plugin list`
+
+Plugin can be removed via `uninstall` command:
+
+`oshino-admin plugin uninstall oshino_statsd`
+
+Making sense of Config
+=======================
+Generated config should look like this:
+
+```yaml
+---
+interval: 10
+riemann:
+  host: localhost
+  port: 5555
+agents:
+  - name: health-check
+    module: oshino.agents.http_agent.HttpAgent
+    url: http://python.org
+    tag: healthcheck
+```
+
+It says, that `interval` at which metrics will be pushed is 10 seconds,
+it is expecting Riemann at `localhost:5555`
+And currently has one Agent called `health-check` which uses included `HttpAgent`
+to do a healthcheck on `http://python.org`. Resulting metrics are tagged `healthcheck`
+
+More info can be found on: [Config](config.md) section.
+
+In general, a proper Riemann's address needs to be providen and array of agents extended
+with agents you require

docs/thirdparty.md

@@ -10,6 +10,3 @@ Known agents:
 - [oshino-hw](https://github.com/CodersOfTheNight/oshino-hw)
 - [oshino-prometheus](https://github.com/CodersOfTheNight/oshino-prometheus)
 
-Creating custom Agent
----------------------
-It can be done by using our [Cookiecutter Template](https://github.com/CodersOfTheNight/oshino-cookiecutter)