Merge pull request #13 from CodersOfTheNight/master

Release 0.2

Author: zaibacu

.travis.yml

@@ -1,6 +1,7 @@
 language: python
 python:
 - 3.5
+- 3.6
 
 install:
   - pip install -r requirements/release.txt

README.md

@@ -26,6 +26,11 @@ How to install
 ==============
 `pip install oshino`
 
+Quickstart
+==========
+It is highly recommended for new users to use [Quickstart Guide](quickstart.md)
+
+
 Riemann. What? Why? How?
 =========================
 Riemann is a backbone of this system. It does alerting, it receives metrics, it aggregates metrics and it decides where to send them (eg.: Graphite, Logstash).

docs/agents.md

@@ -0,0 +1,44 @@
+What is "Agent"?
+-----------------
+In this context, we call a worker which collects specific type of metrics, an agent.
+
+How to use them
+--------------
+To be able to use any of agents, first you must import them.
+Importing is done via config, by adding them under `agents` section
+
+
+An example config could look like this:
+```yaml
+---
+interval: 10
+loglevel: DEBUG
+riemann:
+  host: localhost
+  port: 5555
+  transport: BlankTransport
+agents:
+  - name: echo
+    module: oshino.agents.subprocess_agent.SubprocessAgent
+    script: "echo 'Hello world!'"
+    tag: "bash"
+```
+
+More configuration info can be found under [Config](config.md)
+
+As you can see, there's one agent in agents array, which is called `echo`
+and uses internal `SubprocessAgent` class. 
+This type of agent is able to execute command in command line
+and return it's result as a metric.
+
+Internal agents also includes `HttpAgent` which is able to do HTTP calls,
+and returns response and time it took to execute as metrics.
+
+Usual source of agents is [Third Party agent section](thirdparty.md)
+They can be installed via `pip` command, or using `oshino-admin`. 
+For more information on `oshino-admin` usage, do `oshino-admin --help`.
+
+
+Creating custom Agent
+---------------------
+It can be done by using our [Cookiecutter Template](https://github.com/CodersOfTheNight/oshino-cookiecutter)

docs/augments.md

@@ -0,0 +1,27 @@
+What is "Augment"?
+------------------
+In this context, we a process which reads specific metrics and produces new ones, we call an augment.
+Augment's job is to produce additional insights from metrics we already have.
+
+How to use them?
+----------------
+You need to include specific augment inside config under augments array, very similar to [Agents](agents.md).
+
+An example config could look like this:
+```yaml
+---
+interval: 10
+loglevel: DEBUG
+riemann:
+  host: localhost
+  port: 5555
+  transport: BlankTransport
+augments:
+  - name: moving average
+    key: cpu 
+    module: oshino.augments.stats.SimpleMovingAverage
+    step: 5
+    tag: "sma"
+```
+
+More configuration info can be found under [Config](config.md)

docs/config.md

@@ -1,5 +1,5 @@
 Base Config
------------
+============
 - `interval` - how often to send metrics (in seconds)
 - `riemann.host` - riemann hostname (default: localhost)
 - `riemann.port` - riemann port (default: 5555)
@@ -8,28 +8,47 @@ Base Config
 - `sentry-dsn` - sentry address for error reporting (default: `None`)
 - `executor` - allows to define executor class, default one is `concurrent.futures.ThreadPoolExecutor`
 - `executors-count` - set worker count for executor, default: 10
+- `agents` - an array of agent configs used in this setup
+- `augments` - an array of augment configs used in this setup
+
+Agents
+=======
 
 General Config for Agents
 ------------------------
+- `name`- give name for agent
 - `tag` - adds this tag to all metrics related to this agent
+- `module` - path to module from which actual agent should be loaded. In most cases this path should be declared in actor's README
 
 Http Agent Config
 ----------------
+Agent used to produce HTTP calls to specific website
+module: `oshino.agents.http_agent.HttpAgent`
+
 - `url` - which url to call
 
 Subprocess Agent Config
 -----------------------
+Agent used to execute CLI scripts
+module: `oshino.agents.subprocess_agent.SubprocessAgent`
+
 - `script` - what shell script to execute
 
 
-Suggested Riemann config
--------------------------
-```clojure
- (where (tagged-all ["oshino", "heartbeat"])
-       (changed :state
-         (email "something@something.com")
-       )
-    )
-```
+Augments
+========
+
+General config for Augments
+---------------------------
+- `name`- give name for augment
+- `key` - defines a metric key to listen to
+- `tag` - adds this tag to all metrics related to this augment
+- `module` - path to module from which actual augment should be loaded. In most cases this path should be declared in augment's README
+
+
+Moving Average
+--------------
+Reads metrics and produces moving average
+module: `oshino.augments.stats.MovingAverage`
 
-If Oshino goes down, or encounters some performance hickups - you'll be notified
+- `step` - how much metrics it should inhale before it can produce average

docs/index.md

@@ -1,9 +1,12 @@
 Table of Contents:
 ==================
+- [Quickstart](quickstart.md)
 - [Installiation](install.md)
 - [Riemann](riemann.md)
 - Usage
     - [Events](events.md)
     - [Config](config.md)
 - Extending
+    - [Agents](agents.md)
+    - [Augments](augments.md)
     - [Third Party Agents](thirdparty.md)

docs/quickstart.md

@@ -0,0 +1,106 @@
+Preparing Environment
+======================
+You will need Python version 3.5+ and `pip` installed.
+In the most cases `pip` should come bundled, if building from source, you will need to have `openssl` development package installed, otherwise it will skip.
+
+If for some reason you don't have it and cannot build with `openssl`, there's always:
+
+`wget https://bootstrap.pypa.io/get-pip.py | python`
+
+Virtualenv
+----------
+It is optional, but highly recommended. 
+Can be installed via `pip install virtualenv`
+
+To create actual virtualenv, do:
+`virtualenv venv`
+Or if you have multiple versions of python (usually 2.7 and 3.****), 
+failproof approach would be:
+`python3 -m virtualenv venv`
+
+This will create virtual environment inside folder
+
+Depending on OS, activating environment may slightly differ.
+Linux/MacOS/*BSD it is `source venv/bin/activate`
+Windows: `venv\Scripts\activate`
+
+Installing Oshino
+-----------------
+When everything is prepared, installing is as easy as:
+`pip install oshino`
+
+
+Launching Oshino
+================
+First create config. Easy way to do that is by using `oshino-admin`
+
+`oshino-admin config init config.yaml`
+
+And now you can actually start it:
+`oshino --config=config.yaml`
+
+Running as daemon
+------------------
+To start as a daemon, we need to use `oshino-admin`
+`oshino-admin start --config=config.yaml`
+
+If you're running on your local machine or not as root, it's possible to get error like:
+`Unable to create the pidfile.`
+By default it writes PID (Process ID) file to `/var/run/oshino.pid` and permissions are not always sufficient to do that.
+Issue can be easily mitigated by providing custom path:
+`oshino-admin start --config=config.yaml --pid=oshino.pid`
+
+To check if it's running:
+`oshino-admin status`
+
+To stop it:
+`oshino-admin stop`
+
+Main caveat in using custom pid path, you need to provide it for `status` and `stop` commands as well:
+
+`oshino-admin status --pid=oshino.pid`
+`oshino-admin stop --pid=oshino.pid`
+
+
+Installing plugins
+==================
+
+In the most cases, these plugins are for agents. Agent in Oshino context is metrics collector.
+So for example, if our service is using [Prometheus](https://prometheus.io/) and we want to collect metrics from it, 
+plugin needs to be installed:
+
+`oshino-admin plugin install oshino_prometheus`
+
+To list available plugins:
+`oshino-admin plugin list`
+
+Plugin can be removed via `uninstall` command:
+
+`oshino-admin plugin uninstall oshino_statsd`
+
+Making sense of Config
+=======================
+Generated config should look like this:
+
+```yaml
+---
+interval: 10
+riemann:
+  host: localhost
+  port: 5555
+agents:
+  - name: health-check
+    module: oshino.agents.http_agent.HttpAgent
+    url: http://python.org
+    tag: healthcheck
+```
+
+It says, that `interval` at which metrics will be pushed is 10 seconds,
+it is expecting Riemann at `localhost:5555`
+And currently has one Agent called `health-check` which uses included `HttpAgent`
+to do a healthcheck on `http://python.org`. Resulting metrics are tagged `healthcheck`
+
+More info can be found on: [Config](config.md) section.
+
+In general, a proper Riemann's address needs to be providen and array of agents extended
+with agents you require

docs/thirdparty.md

@@ -10,6 +10,3 @@ Known agents:
 - [oshino-hw](https://github.com/CodersOfTheNight/oshino-hw)
 - [oshino-prometheus](https://github.com/CodersOfTheNight/oshino-prometheus)
 
-Creating custom Agent
----------------------
-It can be done by using our [Cookiecutter Template](https://github.com/CodersOfTheNight/oshino-cookiecutter)

oshino/augments/__init__.py

@@ -0,0 +1,56 @@
+from time import time
+
+from oshino.config import TagMixin
+
+
+class AugmentBase(TagMixin):
+    def __init__(self, data):
+        self._data = data
+
+    def activate(self, client, g):
+        pass
+
+    def is_valid(self):
+        return True
+
+    @property
+    def name(self):
+        return self._data["name"].lower().replace(" ", "-")
+
+    @property
+    def prefix(self):
+        return "{0}.".format(self.name)
+
+    @property
+    def key(self):
+        return self._data["key"]
+
+    def send_event(self, client, **kwargs):
+        tags = [self.tag] if self.tag else []
+        if "tags" in kwargs:
+            for tag in tags:
+                kwargs["tags"].append(tag)
+        else:
+            kwargs["tags"] = tags
+
+        if "time" not in kwargs:
+            kwargs["time"] = int(time())
+
+        client.event(**kwargs)
+
+
+def consume(q):
+    while not q.empty():
+        yield q.get()
+
+
+class InvalidAugment(AugmentBase):
+    """
+    Only for Testing purposes
+    """
+
+    def is_valid(self):
+        return False
+
+    def activate(self, client, g):
+        raise RuntimeError("We shouldn't get here!")

oshino/augments/stats.py

@@ -0,0 +1,36 @@
+from collections import deque
+
+from . import AugmentBase
+
+
+class SimpleMovingAverage(AugmentBase):
+
+    def __init__(self, *args, **kwargs):
+        super(SimpleMovingAverage, self).__init__(*args, **kwargs)
+        self.dq = deque(maxlen=self.step)
+
+    @property
+    def step(self):
+        return self._data.get("step", 5)
+
+    def activate(self, client, g):
+        step = self.step
+
+        stopped = False
+        while not stopped:
+
+            try:
+                event = next(g)
+                self.dq.append(event)
+            except StopIteration:
+                stopped = True
+                break
+
+            data = list(map(lambda x: x["metric_f"], list(self.dq)))
+            if len(data) == step:
+                output = sum(data) / len(data)
+
+                self.send_event(client,
+                                service=self.prefix + "value",
+                                metric_f=output,
+                                state="ok")