Skip to content

Commit 485685f

Browse files
sgrebensgreben
committed
Breaking changes:
- Rename -delta-buffer to -match-buffer - Rename -readjson to -read-json - Rename -jsontemplate to -match-template - Rename -start to -regex / -match-regex - Rename -timeformat to -time-format - Rename .JSONText to .MatchText - Replace .StartText, .StartObject by .Start - Literals (no "{{") given to -match / -match-template are automatically surrounded by {{}} Additions: - Add JSON stream parsing (one-object-per-line format no longer required) - Add -match-condition - Add -time-zone - Add -scale-{sqr,cube,sqrt,cubert} - Add Text and TimeColor preset templates Internal: - Refactor main to avoid duplicating logic between JSON/line input
1 parent 70d0d2b commit 485685f

File tree

7 files changed

+482
-170
lines changed

7 files changed

+482
-170
lines changed

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
VERSION = 5.0.3
1+
VERSION = 6.0.0
22

33
PACKAGES := $(shell go list -f {{.Dir}} ./...)
44
GOFILES := $(addsuffix /*.go,$(PACKAGES))

‎README.md‎

Lines changed: 95 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,14 @@
88
- [Time format](#time-format)
99
- [Template output](#template-output)
1010
- [Color output](#color-output)
11-
- [Stopwatch regex](#stopwatch-regex)
1211
- [JSON input](#json-input)
12+
- [Stopwatch regex](#stopwatch-regex)
13+
- [Stopwatch regex template](#stopwatch-regex-template)
14+
- [Stopwatch condition](#stopwatch-condition)
1315
- [Example](#example)
1416
- [Comments](https://github.com/sgreben/tj/issues/1)
1517

18+
1619
## Get it
1720

1821
Using go get:
@@ -35,25 +38,47 @@ docker pull quay.io/sergey_grebenshchikov/tj
3538

3639
```text
3740
Usage of tj:
38-
-timeformat string
39-
either a go time format string or one of the predefined format names (https://golang.org/pkg/time/#pkg-constants)
4041
-template string
41-
either a go template (https://golang.org/pkg/text/template) or one of the predefined template names
42-
-start string
43-
a regex pattern. if given, only lines matching it (re)start the stopwatch
44-
-readjson
45-
parse each stdin line as JSON
46-
-jsontemplate string
47-
go template, used to extract text from json input. implies -readjson
42+
either a go template (https://golang.org/pkg/text/template) or one of the predefined template names
43+
-time-format string
44+
either a go time format string or one of the predefined format names (https://golang.org/pkg/time/#pkg-constants)
45+
-time-zone string
46+
time zone to use (default "Local")
47+
-match-regex string
48+
a regex pattern. if given, only tokens matching it (re)start the stopwatch
49+
-match-template string
50+
go template, used to extract text used for -match-regex
51+
-match-condition string
52+
go template. if given, only tokens that result in 'true' (re)start the stopwatch
53+
-match-buffer
54+
buffer lines between matches of -match-regex / -match-condition, copy delta values from final line to buffered lines
55+
-match string
56+
alias for -match-template
57+
-condition string
58+
alias for -match-condition
59+
-regex string
60+
alias for -match-regex
61+
-read-json
62+
parse a sequence of JSON objects from stdin
4863
-scale string
49-
either a sequence of hex colors or one of the predefined color scale names (colors go from fast to slow)
50-
(default "BlueToRed")
64+
either a sequence of hex colors or one of the predefined color scale names (colors go from fast to slow)
65+
(default "BlueToRed")
5166
-scale-fast duration
52-
the lower bound for the color scale (default 100ms)
67+
the lower bound for the color scale (default 100ms)
5368
-scale-slow duration
54-
the upper bound for the color scale (default 2s)
55-
-delta-buffer
56-
buffer lines between -start matches, copy delta values from final line to buffered lines
69+
the upper bound for the color scale (default 2s)
70+
-scale-linear
71+
use linear scale (default true)
72+
-scale-cube
73+
use cubic scale
74+
-scale-cubert
75+
use cubic root scale
76+
-scale-sqr
77+
use quadratic scale
78+
-scale-sqrt
79+
use quadratic root scale
80+
-version
81+
print version and exit
5782
```
5883

5984
### JSON output
@@ -65,24 +90,24 @@ $ (echo Hello; echo World) | tj
6590
```
6691

6792
```json
68-
{"timeSecs":1516648762,"timeNanos":1516648762008900882,"time":"2018-01-22T20:19:22+01:00","deltaSecs":0.000015003,"deltaNanos":15003,"delta":"15.003µs","totalSecs":0.000015003,"totalNanos":15003,"total":"15.003µs","text":"Hello"}
69-
{"timeSecs":1516648762,"timeNanos":1516648762009093926,"time":"2018-01-22T20:19:22+01:00","deltaSecs":0.000193044,"deltaNanos":193044,"delta":"193.044µs","totalSecs":0.000208047,"totalNanos":208047,"total":"208.047µs","text":"World"}
93+
{"timeSecs":1517592179,"timeNanos":1517592179895262811,"time":"2018-02-02T18:22:59+01:00","deltaSecs":0.000016485,"deltaNanos":16485,"delta":"16.485µs","totalSecs":0.000016485,"totalNanos":16485,"total":"16.485µs","text":"Hello"}
94+
{"timeSecs":1517592179,"timeNanos":1517592179895451948,"time":"2018-02-02T18:22:59+01:00","deltaSecs":0.000189137,"deltaNanos":189137,"delta":"189.137µs","totalSecs":0.000205622,"totalNanos":205622,"total":"205.622µs","text":"World"}
7095
```
7196

7297
### Time format
7398

74-
You can set the format of the `time` field using the `-timeformat` parameter:
99+
You can set the format of the `time` field using the `-time-format` parameter:
75100

76101
```bash
77-
$ (echo Hello; echo World) | tj -timeformat Kitchen
102+
$ (echo Hello; echo World) | tj -time-format Kitchen
78103
```
79104

80105
```json
81-
{"timeSecs":1516648899,"timeNanos":1516648899954888290,"time":"8:21PM","deltaSecs":0.000012913,"deltaNanos":12913,"delta":"12.913µs","totalSecs":0.000012913,"totalNanos":12913,"total":"12.913µs","text":"Hello"}
82-
{"timeSecs":1516648899,"timeNanos":1516648899955092012,"time":"8:21PM","deltaSecs":0.000203722,"deltaNanos":203722,"delta":"203.722µs","totalSecs":0.000216635,"totalNanos":216635,"total":"216.635µs","text":"World"}
106+
{"timeSecs":1517592194,"timeNanos":1517592194875016639,"time":"6:23PM","deltaSecs":0.000017142,"deltaNanos":17142,"delta":"17.142µs","totalSecs":0.000017142,"totalNanos":17142,"total":"17.142µs","text":"Hello"}
107+
{"timeSecs":1517592194,"timeNanos":1517592194875197515,"time":"6:23PM","deltaSecs":0.000180876,"deltaNanos":180876,"delta":"180.876µs","totalSecs":0.000198018,"totalNanos":198018,"total":"198.018µs","text":"World"}
83108
```
84109

85-
The [constant names from pkg/time](https://golang.org/pkg/time/#pkg-constants) as well as regular go time layouts are admissible values for `-timeformat`:
110+
The [constant names from pkg/time](https://golang.org/pkg/time/#pkg-constants) as well as regular go time layouts are admissible values for `-time-format`:
86111

87112
| Name | Format |
88113
|------------|-------------------------------------|
@@ -115,17 +140,19 @@ $ (echo Hello; echo World) | tj -template '{{ .I }} {{.TimeSecs}} {{.Text}}'
115140
1 1516649679 World
116141
```
117142

118-
The fields available to the template are specified in the [`line` struct](cmd/tj/main.go#L19).
143+
The fields available to the template are specified in the [`token` struct](cmd/tj/main.go#L18).
119144

120145
Some templates are pre-defined and can be used via `-template NAME`:
121146

122-
| Name | Template |
123-
|------------|----------------------------------------------|
124-
| Color | `{{color .}}█{{reset}} {{.Text}}` |
125-
| ColorText | `{{color .}}{{.Text}}{{reset}}` |
126-
| Delta | `{{.DeltaNanos}} {{.Text}}` |
127-
| Time | `{{.TimeString}} {{.Text}}` |
128-
| TimeDelta | `{{.TimeString}} +{{.DeltaNanos}} {{.Text}}` |
147+
| Name | Template |
148+
|------------|--------------------------------------------------|
149+
| Color | `{{color .}}█{{reset}} {{.Text}}` |
150+
| ColorText | `{{color .}}{{.Text}}{{reset}}` |
151+
| Delta | `{{.DeltaNanos}} {{.Text}}` |
152+
| Text | `{{.Text}}` |
153+
| Time | `{{.TimeString}} {{.Text}}` |
154+
| TimeDelta | `{{.TimeString}} +{{.DeltaNanos}} {{.Text}}` |
155+
| TimeColor | `{{.TimeString}} {{color .}}█{{reset}} {{.Text}}`|
129156

130157
### Color output
131158

@@ -149,6 +176,11 @@ The color scale can be set using the parameters `-scale`, `-scale-fast`, and `-
149176
- The `-scale` parameter defines the colors used in the scale.
150177
- The `-scale-fast` and `-scale-slow` parameters define the boundaries of the scale: durations shorter than the value of `-scale-fast` are mapped to the leftmost color, durations longer than the value of `-scale-slow` are mapped to the rightmost color.
151178

179+
The scale is linear by default, but can be transformed:
180+
181+
- `-scale-sqr`, `-scale-sqrt` yields a quadratic (root) scale
182+
- `-scale-cube`, `-scale-cubert` yields a cubic (root) scale
183+
152184
There are several pre-defined color scales:
153185

154186
| Name | Scale |
@@ -165,33 +197,44 @@ There are several pre-defined color scales:
165197

166198
You can also provide your own color scale using the same syntax as the pre-defined ones.
167199

168-
### Stopwatch regex
169-
170-
Sometimes you need to measure the duration between certain *tokens* in the input.
200+
### JSON input
171201

172-
To help with this, `tj` can match each line against a regular expression and only reset the stopwatch (`delta`, `deltaSecs`, `deltaNanos`) when a line matches.
202+
Using `-read-json`, you can tell `tj` to parse stdin as a sequence of JSON objects. The parsed object can be referred to via `.Object`, like this:
173203

174-
The regular expression can be specified via the `-start` parameter.
204+
```bash
205+
$ echo '{"hello": "World"}' | tj -read-json -template "{{.TimeString}} {{.Object.hello}}"
206+
```
175207

176-
### JSON input
208+
```
209+
2018-01-25T21:55:06+01:00 World
210+
```
177211

178-
Using `-readjson`, you can tell `tj` to parse each input line as a separate JSON object. Fields of this object can be referred to via `.Object` in the `line` struct, like this:
212+
The exact JSON string that was parsed can be recovered using `.Text`:
179213

180214
```bash
181-
$ echo '{"hello": "World"}' | tj -readjson -template "{{.TimeString}} {{.Object.hello}}"
215+
$ echo '{"hello" : "World"} { }' | tj -read-json -template "{{.TimeString}} {{.Text}}"
182216
```
183217

184218
```
185-
2018-01-25T21:55:06+01:00 World
219+
2018-01-25T21:55:06+01:00 {"hello" : "World"}
220+
2018-01-25T21:55:06+01:00 { }
186221
```
187222

188-
Additionally, you can also specify a template `-jsontemplate` to extract text from the object. The output of this template is matched against the stopwatch regex.
223+
### Stopwatch regex
224+
225+
Sometimes you need to measure the duration between certain *tokens* in the input.
226+
227+
To help with this, `tj` can match each line against a regular expression and only reset the stopwatch (`delta`, `deltaSecs`, `deltaNanos`) when a line matches. The regular expression can be specified via the `-match-regex` (alias `-regex`) parameter.
228+
229+
### Stopwatch regex template
189230

190-
This allows you to use only specific fields of the object as stopwatch reset triggers. For example:
231+
When using `-match-regex`, you can also specify a template `-match-template` (alias `-match`) to extract text from the current token. The output of this template is matched against the stopwatch regex.
232+
233+
This allows you to use only specific fields of JSON objects as stopwatch reset triggers. For example:
191234

192235
```bash
193236
$ (echo {}; sleep 1; echo {}; sleep 1; echo '{"reset": "yes"}'; echo {}) |
194-
tj -jsontemplate "{{.reset}}" -start yes -template "{{.I}} {{.DeltaNanos}}"
237+
tj -read-json -match .reset -regex yes -template "{{.I}} {{.DeltaNanos}}"
195238
```
196239

197240
```
@@ -201,16 +244,20 @@ $ (echo {}; sleep 1; echo {}; sleep 1; echo '{"reset": "yes"}'; echo {}) |
201244
3 79099
202245
```
203246

204-
The output of the JSON template is stored in the field `.JSONText` of the `line` struct:
247+
The output of the match template is stored in the field `.MatchText` of the `token` struct:
205248

206249
```bash
207-
$ echo '{"message":"hello"}' | tj -jsontemplate "{{.message}}" -template "{{.TimeString}} {{.JSONText}}"
250+
$ echo '{"message":"hello"}' | tj -read-json -match-template .message -template "{{.TimeString}} {{.MatchText}}"
208251
```
209252

210253
```
211254
2018-01-25T22:20:59+01:00 hello
212255
```
213256

257+
### Stopwatch condition
258+
259+
Additionally to `-match-regex`, you can specify a `-match-condition` go template. If this template produces the literal string `true`, the stopwatch is reset - "matches" of the `-match-condition` are treated like matches of the `-match-regex`.
260+
214261
## Example
215262

216263
Finding the slowest step in a `docker build` (using `jq`):
@@ -225,8 +272,8 @@ RUN echo Done being slow
225272

226273
```bash
227274
$ docker build . |
228-
tj -start ^Step |
229-
jq -s 'max_by(.deltaNanos) | {step:.startText, duration:.delta}'
275+
tj -regex ^Step |
276+
jq -s 'max_by(.deltaNanos) | {step:.start.text, duration:.delta}'
230277
```
231278

232279
```json
@@ -237,7 +284,7 @@ Alternatively, using color output and buffering:
237284

238285
```bash
239286
$ docker build . |
240-
tj -start ^Step -template Color -scale GreenToGreenToRed -delta-buffer
287+
tj -regex ^Step -match-buffer -template Color -scale-cube
241288
```
242289

243290
![Docker build with color output](docs/images/docker.png)

‎cmd/tj/json.go‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
package main
2+
3+
import (
4+
"bytes"
5+
"encoding/json"
6+
"io"
7+
"os"
8+
"strings"
9+
)
10+
11+
const jsonStreamScratchBufferBytes = 4096
12+
13+
type jsonStream struct {
14+
token
15+
Text string `json:"-"` // the original text that Object was parsed from
16+
Object interface{} `json:"object,omitempty"`
17+
18+
textBuffer *bytes.Buffer // intercepts bytes read by decoder
19+
scratchBuffer []byte // determines size of decoder.Buffered()
20+
buffer *tokenBuffer
21+
decoder *json.Decoder
22+
decodeError error
23+
done bool
24+
}
25+
26+
func newJSONStream() *jsonStream {
27+
textBuffer := bytes.NewBuffer(nil)
28+
tee := io.TeeReader(os.Stdin, textBuffer)
29+
return &jsonStream{
30+
decoder: json.NewDecoder(tee),
31+
textBuffer: textBuffer,
32+
scratchBuffer: make([]byte, jsonStreamScratchBufferBytes),
33+
buffer: &tokenBuffer{},
34+
}
35+
}
36+
37+
func (j *jsonStream) Token() *token {
38+
return &j.token
39+
}
40+
41+
func (j *jsonStream) CopyCurrent() tokenStream {
42+
return &jsonStream{
43+
token: j.token,
44+
Object: j.Object,
45+
}
46+
}
47+
48+
func (j *jsonStream) AppendCurrentToBuffer() {
49+
*j.buffer = append(*j.buffer, j.CopyCurrent())
50+
}
51+
52+
func (j *jsonStream) FlushBuffer() {
53+
j.buffer.flush(j)
54+
}
55+
56+
func (j *jsonStream) CurrentMatchText() string {
57+
if matchTemplate != nil {
58+
return matchTemplate.execute(j.Object)
59+
}
60+
return j.Text
61+
}
62+
63+
func (j *jsonStream) Err() error {
64+
if j.decodeError == io.EOF {
65+
return nil
66+
}
67+
return j.decodeError
68+
}
69+
70+
func (j *jsonStream) readerSize(r io.Reader) int {
71+
total := 0
72+
var err error
73+
var n int
74+
for err == nil {
75+
n, err = r.Read(j.scratchBuffer)
76+
total += n
77+
}
78+
return total
79+
}
80+
81+
func (j *jsonStream) Scan() bool {
82+
j.Object = new(interface{})
83+
err := j.decoder.Decode(&j.Object)
84+
numBytesNotParsedByJSON := j.readerSize(j.decoder.Buffered()) // "{..} XYZ" -> len("XYZ")
85+
bytesUnreadByUs := j.textBuffer.Bytes() // "{..} XYZ" -> "{..} XYZ"
86+
numBytesUnreadByUs := len(bytesUnreadByUs)
87+
numBytesParsedByJSON := numBytesUnreadByUs - numBytesNotParsedByJSON // len("{..}")
88+
bytesReadByJSON := bytesUnreadByUs[:numBytesParsedByJSON] // "{..} XYZ" -> "{..}"
89+
j.Text = strings.TrimSpace(string(bytesReadByJSON))
90+
j.textBuffer.Next(numBytesParsedByJSON) // "*{..} XYZ" -> "*XYZ"
91+
if err != nil {
92+
if j.decodeError == nil || j.decodeError == io.EOF {
93+
j.decodeError = err
94+
}
95+
return false
96+
}
97+
return true
98+
}

0 commit comments

Comments
 (0)