|
2 | 2 |
|
3 | 3 | Ship logs from fly to other providers using [NATS](https://docs.nats.io/) and [Vector](https://vector.dev/) |
4 | 4 |
|
5 | | -Here we have some vector configs, alongside a wrapper script to run it all, |
6 | | -that will subscribe to a log stream of your organisation's logs and ship it to various providers. |
| 5 | +In this repo you will find various [Vector Sinks](https://vector.dev/docs/reference/configuration/sinks/) along with the required fly config. The end result is a Fly.IO application that automatically reads your organisation logs and sends them to external providers. |
7 | 6 |
|
8 | | -# Configuration |
| 7 | +# Quick start |
9 | 8 |
|
10 | | -Create a new Fly app based on this [Dockerfile](./Dockerfile) and configure using the following secrets: |
| 9 | +1. Create a new fly logger app based on our docker image |
| 10 | + |
| 11 | +``` |
| 12 | +fly launch --image ghcr.io/superfly/fly-log-shipper:latest |
| 13 | +``` |
| 14 | + |
| 15 | +2. Set [NATS source secrets](#nats-source-configuration) for your new app |
| 16 | +3. Set your desired [provider](#provider-configuration) from below |
| 17 | + |
| 18 | +**Thats it** - no need to setup NATs clients within your apps, as fly apps are already sending monitoring information back to fly which we can read. |
| 19 | + |
| 20 | +However for advanced uses you can still configure a NATs client in your apps to talk to this NATs server. See [NATS](#nats) |
11 | 21 |
|
12 | 22 | ## NATS source configuration |
13 | 23 |
|
14 | 24 | | Secret | Description | |
15 | | -| -------------- |------------------------------------------------------------------------------------------------------------------| |
| 25 | +| -------------- | ---------------------------------------------------------------------------------------------------------------- | |
16 | 26 | | `ORG` | Organisation slug (default to `personal`) | |
17 | 27 | | `ACCESS_TOKEN` | Fly personal access token (required; set with `fly secrets set ACCESS_TOKEN=$(fly auth token)`) | |
18 | 28 | | `SUBJECT` | Subject to subscribe to. See [[NATS]] below (defaults to `logs.>`) | |
@@ -139,16 +149,52 @@ One of these is required for New Relic logs. New Relic recommend the license key |
139 | 149 |
|
140 | 150 | # NATS |
141 | 151 |
|
142 | | -The log stream is provided through the [NATS protocol](https://docs.nats.io/nats-protocol/nats-protocol) |
143 | | -and is limited to subscriptions to logs in your organisations. |
144 | | -The NATS source takes some Fly specific environment variables to connect to the stream, |
145 | | -but any NATS client can connect to `fdaa::3` on port `4223` in a Fly vm, |
146 | | -with an organisation slug as the username and a Fly Personal Access Token as the password. |
| 152 | +The log stream is provided through the [NATS protocol](https://docs.nats.io/nats-protocol/nats-protocol) and is limited to subscriptions to logs in your organisations. |
| 153 | + |
| 154 | +## Connecting |
| 155 | + |
| 156 | +> Note: You do **not** have to manually connect a NAT Client, see [Quick Start](#quick-start) |
| 157 | +
|
| 158 | +If you want to add custom behaviours or modify the subject sent from your app, then you can connect your app to the NATs server manually. |
| 159 | + |
| 160 | +Any fly app can connect to the NATs server on `nats://[fdaa::3]:4223` (IPV6). |
| 161 | + |
| 162 | +**Note: you will need to supply a user / password.** |
| 163 | + |
| 164 | +> **User**: is your Fly organisation slug, which you can obtain from `fly orgs list` > **Pass**: is your fly token, which you can obtain from `fly auth token` |
| 165 | +
|
| 166 | +#### Example using the NATs client |
| 167 | + |
| 168 | +Launch a nats client based on the nats-server image |
| 169 | + |
| 170 | +``` |
| 171 | +fly launch --image="synadia/nats-server:nightly" --name="nats-client" |
| 172 | +``` |
| 173 | + |
| 174 | +SSH into the new app |
| 175 | + |
| 176 | +``` |
| 177 | +fly -a nats-client ssh console |
| 178 | +``` |
| 179 | + |
| 180 | +``` |
| 181 | +nats context add nats --server [fdaa::3]:4223 --description "NATS Demo" --select \ |
| 182 | + --user <YOUR FLY ORG SLUG> |
| 183 | + --pass <YOUR PAT> |
| 184 | +``` |
| 185 | + |
| 186 | +``` |
| 187 | +nats pub "logs.test" "hello world" |
| 188 | +``` |
| 189 | + |
| 190 | +## Subject |
147 | 191 |
|
148 | 192 | The subject schema is `logs.<app_name>.<region>.<instance_id>` and the standard |
149 | 193 | [NATS wildcards](https://docs.nats.io/nats-concepts/subjects#wildcards) can be used. |
150 | 194 | In this app, the `SUBJECT` secret can be used to set the subject and limit the scope of the logs streamed. |
151 | 195 |
|
| 196 | +## Queue |
| 197 | + |
152 | 198 | If you would like to run multiple vm's for high availability, the NATS endpoint supports |
153 | 199 | [subscription queues](https://docs.nats.io/nats-concepts/queue) to ensure messages are only sent to one |
154 | 200 | subscriber of the named queue. The `QUEUE` secret can be set to configure a queue name for the client. |
|
0 commit comments