Skip to content

Commit eef0354

Browse files
qnikstqnikst
committed
Improve readme.
1 parent 066e6d5 commit eef0354

File tree

3 files changed

+221
-19
lines changed

3 files changed

+221
-19
lines changed

‎Readme.markdown‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,12 @@
1-
# Cheops-logger
1+
# Cheops-logger - logging system.
22

3-
* co-log-json - structured logging library
3+
This repostory provide two packages
44

5-
* cheops-logger - full featured logging library
5+
* [co-log-json](./co-log-json) — structured logging library. This is a general purpose library
6+
that can be combined with the control structures of your application and it does not introduce
7+
unnessesary dependencies.
8+
9+
* [cheops-logger](./cheops-logger) — wrapper over a set of logging and related libraries.
10+
This library is ready to be used in application. It provides a set of reasonable defaults and choices
11+
and is used in internal applications.
612

‎co-log-json/Readme.md‎

Lines changed: 212 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,13 @@
22

33
`co-log-json` allows to write structured logs in your application.
44

5-
With this library structured means that each log message can contain additional
6-
context that is structured and can be easily parsed by the external system.
7-
8-
That context forms a nested scope, and all messages in the score inherit provided
5+
The library allows to add additional machine readable context to the log
6+
messages. I.e. each log message is a json object with a predefined structure
7+
where additional user context can be added. Such logs can be easily indexed
8+
by external logs systems such as [graylog](https://www.graylog.org) or
9+
[Elastic search](https://www.elastic.co). The message logging is done using
10+
a special context, that keeps information about additional attributes.
11+
Such contexts forms a nested scope, and all messages in the score inherit provided
912
context:
1013

1114
```
@@ -18,21 +21,216 @@ context:
1821
| ==> {"message": "user entered the door", "data": { "user": "Joe", "tag": "human"}
1922
```
2023

24+
2125
In other words this library takes two choices:
2226

23-
1. We write full context with each message. This approach allows to use external
24-
tools (like elastic-search) that can decode message without any additional work.
25-
Any message can be lost without affecting ability to decode other messages. Another
26-
choice there could be emitting a list of events (possibly as a bytecode), such approach
27-
is taken by the tracing libraries. But this approach require more works during storing
28-
(or reading) logs, and in case in some logs are logs may render later ones unusable or
29-
lossing required info.
27+
1. We write full context with each message.
28+
This approach allows to use external tools (like elastic-search) that can index message
29+
without any additional work and information.
30+
Any message can be lost without affecting ability to decode other messages. Alternative
31+
approach could be emitting a list of events (possibly as a bytecode), such approach
32+
is taken by the tracing libraries. It requires more works during reading and indexing
33+
logs, and in case in some logs are logs may render later ones unusable or lose some info.
3034

3135
2. We keep a state context so we don't need to attach context messages to each one.
32-
The contrary approach is extraction of the structure information from the message itself.
33-
In that approach all interesting data should present in each message.
36+
Alternative approach is extraction of the structure information from the message itself.
37+
This approach is taken in some structured logging libraries, it can provide better error
38+
messages. However it requires to think about the message context all the time (and developer
39+
user may not even know all interesting context where the message will be used).
40+
3441

3542
# Using a library
3643

37-
TODO
44+
In order to use a library you'll need to add `co-log-json` to the dependency list.
45+
46+
```
47+
library.cabal
48+
49+
library
50+
build-depends:
51+
base,
52+
co-log-json ^>= 0.0
53+
```
54+
55+
56+
## Setting the library
57+
58+
59+
```haskell
60+
import Colog.Json -- Core API for structured logging
61+
import Colog.Json.Action (logToHandle) -- Actions to store the message
62+
import System.IO (stderr)
63+
64+
main :: IO ()
65+
main = do
66+
-- First we need to setup log storing function:
67+
-- In order to emit logs we need to create a context.
68+
-- Context takes the action to store log and it's possible to
69+
-- attach additional information to it.
70+
71+
let context = mkLogger (logToHandle stderr)
72+
-- ^
73+
-- |
74+
-- +------ `LogAction IO Message`
75+
-- (see discussion "Why IO below")
76+
--
77+
-- Once 'context' is created it can be passed to the other parts
78+
-- of the code or used to emit logs.
79+
80+
logDebug context "Hi, there!"
81+
-- ^ ^
82+
-- | |
83+
-- | +----- `LogStr` type - is an efficient text builder.
84+
-- |
85+
-- |
86+
-- +-- log context
87+
88+
-- Will output:
89+
--
90+
-- ```
91+
-- {"severity":"DEBUG", "thread": 1, "message": "Hi, there!"}
92+
-- ```
93+
-- ^ ^ ^
94+
-- | | +------------ Message itself
95+
-- | +----------------------------- Id of the thread that emitted the message
96+
-- +----------------- Severity level: Debug, Info, Notice, Error, Critical, Alert, Emergency
97+
--
98+
-- to the stderr.
99+
100+
-- There are other helper functions list `logInfo`, `logNotice`, etc.
101+
```
102+
103+
Now let's discuss `LogStr` type. It exists in order to efficiently generate message without
104+
extra allocations that can be avoided. Basically it's just a [Text.Builder](http://hackage.haskell.org/package/text-1.2.4.0/docs/Data-Text-Lazy-Builder.html) though iternal
105+
representation may evolve in the future. `LogStr` allows message concatenation without allocating
106+
intermediate structures, so it allows efficient log building. For example line:
107+
108+
```haskell
109+
logDebug context $ "a" <> "b" <> "c"
110+
```
111+
112+
will write all text string "a", "b", "c" directy to the buffer, and can actually do more optimizations.
113+
So it's quite safe to use the librart even with a large amount of logs.
114+
115+
The 'LogStr' type implement [IsString](http://hackage.haskell.org/package/base-4.14.0.0/docs/Data-String.html#t:IsString) instance it means that you can just write string literals
116+
and they will be treated as 'LogStr'. To concatenate two 'LogStr' you can use [`<>`](https://hackage.haskell.org/package/base-4.14.0.0/docs/Data-Semigroup.html#t:Semigroup) operation.
117+
118+
In addition you can convert any string lines that has [StringConv a T.Text](https://hackage.haskell.org/package/string-conv-0.1.2/docs/Data-String-Conv.html), i.e. can be converted
119+
to LogStr using `ls :: StringConv a T.Text => a -> LogStr`.
120+
Efficiency and safety of this convertion depends on the concrete instance implementation and is out
121+
of control of the package.
122+
123+
```haskell
124+
logDebug context $ "foo is " <> ls ("ы":: ByteString)
125+
```
126+
127+
In case your data structure does not implement `StringConv a` it's possible to use not efficient but
128+
very general `showLS :: Show a => a -> LogStr` method that will work for any type that has Show instance.
129+
130+
## Adding context.
131+
132+
But just writing json messages is not very interesting, we want to control the context of the message.
133+
In order to add user data to the context we can use `addContext :: PushContext -> LoggerEnv -> LoggerEnv`
134+
method.
135+
136+
```haskell
137+
138+
let context1 = addContext (sl "user_id" (1::Int)) context
139+
-- ^ ^ ^ ^ ^
140+
-- | | | | +--- old context
141+
-- | | | +------------ user data
142+
-- | | +---------------------- key for the user data
143+
-- | +---------------------------- function that generates an entry
144+
-- +------------------------------------------------- new context that has data attached
145+
146+
logDebug context1 "Hi again!"
147+
148+
-- Will emit to the stderr:
149+
--
150+
-- ```
151+
-- {"severity":"DEBUG", "thread": 1, "message": "Hi, there!", "data": {"user_id":1}}
152+
-- ```
153+
-- ^ ^ ^
154+
-- | | +-- value
155+
-- | +------- key
156+
-- +--- all user data is kept under 'data' key
157+
```
158+
159+
Sidenote: 'sl' function looks redundant, but it's needed in order to provide a future
160+
compatibility for a case when internal structure of user data is changing. Historically
161+
it came from [katip](https://hackage.haskell.org/package/katip) library, there are more functions that allows context creation but
162+
163+
Function `sl :: ToJSON a => T.Text -> a -> PushContext` prepares update to the context.
164+
It encodes user data to JSON. This is why we had to add type annotation to `1` otherwise GHC had
165+
no means to infer the type).
166+
167+
It's important that library does not perform any compacation of key-values, i.e.:
168+
169+
```
170+
logDebug (addContext (sl "user_id" 2) context1) "Who am I?"
171+
-- Will emit:
172+
-- ```
173+
-- {"severity":"DEBUG", "thread": 1, "message": "Hi, there!", "data": {"user_id":1, "user_id": 2}}
174+
-- ```
175+
```
176+
177+
This is done for purpose, but may be a subject to change in the future major versions.
178+
179+
`addContext` is not the only function that allows to modify context, there is the other one
180+
`addNamespace :: Text -> LoggerEnv -> LoggerEnv`, this function adds an entry about the
181+
current namespace. Namespace is a nested list of strings that tells the logical part of
182+
the codebase the log belongs to. It's kept separate fom the data, and it's possible to
183+
use in library filtering (using [cfilter](https://kowainik.github.io/posts/2018-09-25-co-log#cfilter)) or filtering in external system.
184+
185+
186+
The described functionality is enough to perform logging. However it may worth discussing
187+
interporability with the rest of co-log ecosystem and ergonomics.
188+
189+
## co-log interporability
190+
191+
co-log ecosystem works with "LogAction m a" and in the package we use "LogEnv" it means
192+
that we are losing most of the benefits of the library and can't use a lot of utility
193+
functions. In order to improve the situation it's possible to convert LogEnv into
194+
`LogAction m (Severity,LogStr)` using function `unLogger`. Then we will have log action
195+
that will emit a message with the current context, but context will no longer be modifiable.
196+
197+
# Ergonomics
198+
199+
The package provides no ergonomic tools by default and it's important that `addContext` is a pure
200+
function. On the one hand it introduces a log of boilerplate code but on the other.
201+
202+
It allows to modify context even outside of the effectful computation for example with servant you may have:
203+
204+
```
205+
-- | Catalogue of items
206+
data CatalogueApi route = CatalogueApi
207+
{ _the_catalogue :: route :- Capture "catalogue_id" CatalogueId :> ToServantAPI Bar
208+
, ...
209+
}
210+
211+
-- Item
212+
data ItemApi route = ItemApi
213+
{ _items :: route :- "items" :> Get '[JSON] (Vector Item)
214+
, ...
215+
}
216+
217+
handleCatalogue :: LoggerEnv -> ToServant CatalogueApi AsServer
218+
handleCatalogue ctx' = CatalogueApi
219+
{ _the_catalogue = \catalogue_id ->
220+
handleItem (addContext "catalogue_id" catalogue_id) ctx) catalogue_id
221+
, ...
222+
}
223+
where
224+
ctx = addNamespace "catalogue" ctx'
225+
226+
handleItem :: LoggerEnv -> CatalogueId -> ToServant ItemApi AsServer
227+
handleItem ctx' catalogue_id = ItemApi ...
228+
where ctx = addNamespace "item" ctx'
229+
```
230+
231+
So we can build computation that we will execute later while modifying the context.
38232

233+
On the other hand it we can always wrap this pure function into an effect system
234+
of choice be it either handle pattern or readert or mtl or free-monad or effect system.
235+
This library does not target any particular solution and do not want to introduce
236+
neither additional dependencies nor additional restrictions on the user.

‎examples/cheops-logger-example.cabal‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,6 @@ build-type: Simple
1616
executable Main
1717
main-is: Main.hs
1818
default-language: Haskell2010
19-
exposed-modules:
20-
Cheops.Logger
2119
build-depends:
2220
base >= 4.11 && <5
2321
, cheops-logger

0 commit comments

Comments
 (0)