|
| 1 | + |
| 2 | +Tmux sysstat plugin |
| 3 | +=================== |
| 4 | +Allow to print CPU usage, memory & swap, load average, net I/O metrics in Tmux status bar |
| 5 | + |
| 6 | + |
| 7 | + |
| 8 | +You might checkout [tmux-config](https://github.com/samoshkin/tmux-config) repo to see this plugin in action. |
| 9 | + |
| 10 | +Features |
| 11 | +-------- |
| 12 | +- CPU usage |
| 13 | +- Memory available/free, used, total (KiB,MiB,GiB), free/used % |
| 14 | +- Swap used, free, total, free/used % |
| 15 | +- load average for last 1,5,15 minutes |
| 16 | +- configurable thresholds (low, medium, stress) with custom colors |
| 17 | +- tweak each metric output using templates (e.g, 'used 10% out of 16G') |
| 18 | +- configurable size scale (K,M,G) |
| 19 | +- OSX, Linux support |
| 20 | +- [ ] **TODO:** network I/O metric support |
| 21 | + |
| 22 | +Tested on: OS X El Capitan 10.11.5, Ubuntu 14 LTS, CentOS 7 |
| 23 | + |
| 24 | + |
| 25 | +Installation |
| 26 | +------------ |
| 27 | +Best installed through [Tmux Plugin Manager](https://github.com/tmux-plugins/tpm) (TMP). Add following line to your `.tmux.conf` file: |
| 28 | + |
| 29 | +``` |
| 30 | +set -g @plugin 'samoshkin/tmux-plugin-sysstat' |
| 31 | +``` |
| 32 | + |
| 33 | +Use `prefix + I` from inside tmux to install all plugins and source them. If you prefer, same effect can be achieved from [command line](https://github.com/tmux-plugins/tpm/blob/master/docs/managing_plugins_via_cmd_line.md): |
| 34 | + |
| 35 | +``` |
| 36 | +$ ~.tmux/plugins/tpm/bin/install_plugins |
| 37 | +``` |
| 38 | + |
| 39 | +Basic usage |
| 40 | +----------- |
| 41 | + |
| 42 | +Once plugged in, tmux `status-left` or `status-right` options can be configured with following placeholders. Each placeholder will be expanded to metric's default output. |
| 43 | + |
| 44 | +- `#{sysstat_cpu}`, CPU usage - `CPU:40.2%` |
| 45 | +- `#{sysstat_mem}`, memory usage - `MEM:73%` |
| 46 | +- `#{sysstat_swap}`, swap usage - `SW:66%` |
| 47 | +- `#{sysstat_loadavg}`, system load average - `0.25 0.04 0.34` |
| 48 | + |
| 49 | +For example: |
| 50 | +``` |
| 51 | +set -g status-right "#{sysstat_cpu} | #{sysstat_mem} | #{sysstat_swap} | #{sysstat_loadavg} | #[fg=cyan]#(echo $USER)#[default]@#H" |
| 52 | +``` |
| 53 | + |
| 54 | +Changing default output |
| 55 | +------------------------ |
| 56 | + |
| 57 | +You can change default output for CPU and memory metrics, if you need more fields to show, or you want to provide custom template. In your `.tmux.conf`: |
| 58 | + |
| 59 | +For example, to get `Used 4.5G out of 16G` output for memory metric: |
| 60 | + |
| 61 | +``` |
| 62 | +set -g @sysstat_mem_view_tmpl '#Used [fg=#{mem.color}]#{mem.used}#[default] out of #{mem.total}' |
| 63 | +``` |
| 64 | + |
| 65 | +If you don't want `CPU:` prefix and don't like colored output for CPU metric: |
| 66 | + |
| 67 | +``` |
| 68 | +set -g @sysstat_cpu_view_tmpl '#{cpu.pused}' |
| 69 | +``` |
| 70 | + |
| 71 | +### Supported fields |
| 72 | + |
| 73 | +As you can see, each metric can be configured with template, containing fixed text (`CPU:`), color placeholder (`#[fg=#{mem.color}]`) and field placeholder (`#{mem.used}`). This approach gives you the ultimate control over the output for each metric. Following field placeholders are supported: |
| 74 | + |
| 75 | +<table> |
| 76 | + <th>CPU</th> |
| 77 | + <tr> |
| 78 | + <td><code>#{cpu.color}</code></td> |
| 79 | + <td>main metric color</td> |
| 80 | + </tr> |
| 81 | + <tr> |
| 82 | + <td><code>#{cpu.pused}</code></td> |
| 83 | + <td>CPU usage percentage</td> |
| 84 | + </tr> |
| 85 | +</table> |
| 86 | + |
| 87 | +<table> |
| 88 | + <th>Memory</th> |
| 89 | + <tr> |
| 90 | + <td><code>#{mem.color}</code></td> |
| 91 | + <td>main metric color</td> |
| 92 | + </tr> |
| 93 | + <tr> |
| 94 | + <td><code>#{mem.free}</code></td> |
| 95 | + <td>free/available memory</td> |
| 96 | + </tr> |
| 97 | + <tr> |
| 98 | + <td><code>#{mem.pfree}</code></td> |
| 99 | + <td>free memory percentage against total</td> |
| 100 | + </tr> |
| 101 | + <tr> |
| 102 | + <td><code>#{mem.used}</code></td> |
| 103 | + <td>used memory</td> |
| 104 | + </tr> |
| 105 | + <tr> |
| 106 | + <td><code>#{mem.pused}</code></td> |
| 107 | + <td>used memory percentage against total</td> |
| 108 | + </tr> |
| 109 | + <tr> |
| 110 | + <td><code>#{mem.total}</code></td> |
| 111 | + <td>total installed memory</td> |
| 112 | + </tr> |
| 113 | +</table> |
| 114 | + |
| 115 | +<table> |
| 116 | + <th>Swap</th> |
| 117 | + <tr> |
| 118 | + <td><code>#{swap.color}</code></td> |
| 119 | + <td>main swap metric color</td> |
| 120 | + </tr> |
| 121 | + <tr> |
| 122 | + <td><code>#{swap.free}</code></td> |
| 123 | + <td>free swap memory</td> |
| 124 | + </tr> |
| 125 | + <tr> |
| 126 | + <td><code>#{swap.pfree}</code></td> |
| 127 | + <td>free swap memory percentage against total swap space</td> |
| 128 | + </tr> |
| 129 | + <tr> |
| 130 | + <td><code>#{swap.used}</code></td> |
| 131 | + <td>used swap memory</td> |
| 132 | + </tr> |
| 133 | + <tr> |
| 134 | + <td><code>#{swap.pused}</code></td> |
| 135 | + <td>used swap memory percentage against total swap space</td> |
| 136 | + </tr> |
| 137 | + <tr> |
| 138 | + <td><code>#{swap.total}</code></td> |
| 139 | + <td>total swap space</td> |
| 140 | + </tr> |
| 141 | +</table> |
| 142 | + |
| 143 | +### Change size scale |
| 144 | + |
| 145 | +free/used/total memory can be shown both in absolute and relative units. When it comes to absolute units, you can choose *size scale factor* to choose between GiB, MiB, KiB. Default is GiB. If you have less than 3-4G memory installed, it makes sense to use MiB. KiB option is less practical, because it yields pretty lengthy output, which does not fit status bar limited estate. |
| 146 | + |
| 147 | +``` |
| 148 | +set -g @sysstat_mem_size_unit "G" |
| 149 | +``` |
| 150 | + |
| 151 | +If you choose `G` for size scale, output will have `%.1f` (1 digit after floating point), otherwise size is integer (4.5G, 1024M, 1232345K). |
| 152 | + |
| 153 | +Thresholds and colored output |
| 154 | +--------------- |
| 155 | +Each metric output is colored by default. Colors vary depending on metric value. |
| 156 | + |
| 157 | +<table> |
| 158 | + <tr> |
| 159 | + <td><b>Threshold</b></td> |
| 160 | + <td><b>CPU</b></td> |
| 161 | + <td><b>Memory</b></td> |
| 162 | + <td><b>Swap</b></td> |
| 163 | + <td><b>Default color</b></td> |
| 164 | + </tr> |
| 165 | + <tr> |
| 166 | + <td>low</td> |
| 167 | + <td>x < 30%</td> |
| 168 | + <td>x < 75%</td> |
| 169 | + <td>x < 25%</td> |
| 170 | + <td>green</td> |
| 171 | + </tr> |
| 172 | + <tr> |
| 173 | + <td>medium</td> |
| 174 | + <td>30% < x < 80%</td> |
| 175 | + <td>75% < x < 90%</td> |
| 176 | + <td>25% < x < 75%</td> |
| 177 | + <td>yellow</td> |
| 178 | + </tr> |
| 179 | + <tr> |
| 180 | + <td>high</td> |
| 181 | + <td>x > 80%</td> |
| 182 | + <td>x > 90%</td> |
| 183 | + <td>x > 75%</td> |
| 184 | + <td>red</td> |
| 185 | + </tr> |
| 186 | +</table> |
| 187 | + |
| 188 | +You can change thresholds in your `.tmux.conf`: |
| 189 | + |
| 190 | +``` |
| 191 | +set -g @sysstat_cpu_medium_threshold "75" |
| 192 | +set -g @sysstat_cpu_stress_threshold "95" |
| 193 | +
|
| 194 | +set -g @sysstat_mem_medium_threshold "85" |
| 195 | +set -g @sysstat_mem_stress_threshold "95" |
| 196 | +
|
| 197 | +set -g @sysstat_swap_medium_threshold "80" |
| 198 | +set -g @sysstat_swap_stress_threshold "90" |
| 199 | +``` |
| 200 | + |
| 201 | +You can change colors for each threshold individually. You can use ANSI basic colors (red, cyan, green) or if your terminal supports 256 colors (and most do nowadays), use `colourXXX` format. |
| 202 | + |
| 203 | +``` |
| 204 | +set -g @sysstat_cpu_color_low "colour076" |
| 205 | +set -g @sysstat_cpu_color_medium "colour220" |
| 206 | +set -g @sysstat_cpu_color_stress "colour160" |
| 207 | +set -g @sysstat_mem_color_low "green" |
| 208 | +set -g @sysstat_mem_color_medium "blue" |
| 209 | +set -g @sysstat_mem_color_stress "cyan" |
| 210 | +``` |
| 211 | + |
| 212 | +`#{(mem|cpu|swap).color}` placeholder in your `@sysstat_(mem|cpu|swap)_view_tmpl` would be replaced by corresponding color, depending on whether metric value falls in particular threshold. |
| 213 | + |
| 214 | +### 256 color palette support |
| 215 | + |
| 216 | +For 256 color palette support, make sure that `tmux` and parent terminal are configured with correct terminal type. See [here](https://unix.stackexchange.com/questions/1045/getting-256-colors-to-work-in-tmux) and [there](https://github.com/tmux/tmux/wiki/FAQ) |
| 217 | + |
| 218 | +``` |
| 219 | +# ~/.tmux.conf |
| 220 | +set -g default-terminal "screen-256color" |
| 221 | +``` |
| 222 | + |
| 223 | +``` |
| 224 | +# parent terminal |
| 225 | +$ echo $TERM |
| 226 | +xterm-256color |
| 227 | +
|
| 228 | +# jump into a tmux session |
| 229 | +$ tmux new |
| 230 | +$ echo $TERM |
| 231 | +screen-256color |
| 232 | +``` |
| 233 | + |
| 234 | + |
| 235 | + |
| 236 | +### Multiple colors for each threshold |
| 237 | + |
| 238 | +You can have up to *3* colors configured for each threshold. To understand why you might need this, let tackle this task. Note, this is rather advanced use case. |
| 239 | + |
| 240 | +> I want `CPU: #{cpu.pused}` metric output, have green and yellow text colors at "low" and "medium" threshold, and finally, for "high" threshold, I want to use red color, but reverse foreground and background, that is use red for background, and white for text. More over I want "CPU:" text colored apart in red |
| 241 | +
|
| 242 | +Like this: |
| 243 | + |
| 244 | + |
| 245 | + |
| 246 | +You can achieve the result using following configuration: |
| 247 | + |
| 248 | +``` |
| 249 | +set -g @sysstat_cpu_view_tmpl '#[fg=#{cpu.color3}]CPU:#[default] #[fg=#{cpu.color},bg=#{cpu.color2}]#{cpu.pused}#[default]' |
| 250 | +
|
| 251 | +set -g @sysstat_cpu_color_low "$color_level_ok default default" |
| 252 | +set -g @sysstat_cpu_color_medium "$color_level_warn default default" |
| 253 | +set -g @sysstat_cpu_color_stress "white,bold $color_level_stress $color_level_stress" |
| 254 | +``` |
| 255 | + |
| 256 | +Tmux status-interval setting |
| 257 | +----------------------------- |
| 258 | +You can configure status refresh interval, increasing or reducing frequency of `tmux-plugin-sysstat` command invocations. |
| 259 | + |
| 260 | +``` |
| 261 | +set -g status-interval 5 |
| 262 | +``` |
| 263 | + |
| 264 | +It's adviced to set `status-interval` to some reasonable value, like 5-10 seconds. More frequent updates (1 second) are useless, because they distract, and results in extra resource stress spent on metrics calculation itself. |
| 265 | + |
| 266 | + |
| 267 | + |
| 268 | +Internals: CPU calculation |
| 269 | +-------------------------------------------------- |
| 270 | +<span style="color: blue">**NOTE:** Stop here if you want to just use this plugin without making your feet wet. If you're hardcore tmux user and are curious about internals, keep reading</span> |
| 271 | + |
| 272 | +Internally, we use `iostat` and `top` on OSX, and `vmstat` and `top` on Linux to collect metric value. Neither requires you to install extra packages. These commands are run in sampling mode to report stats every N seconds M times. First sample include average values since the system start. Second one is the average CPU per second for last N seconds (exactly what we need) |
| 273 | + |
| 274 | +For example: |
| 275 | + |
| 276 | +``` |
| 277 | +$ iostat -c 2 -w 5 |
| 278 | + disk0 cpu load average |
| 279 | + KB/t tps MB/s us sy id 1m 5m 15m |
| 280 | + 44.22 6 0.26 3 2 95 1.74 1.90 2.15 |
| 281 | + 5.47 8 0.04 4 5 91 1.84 1.92 2.16 << use this row, 2nd sample |
| 282 | +``` |
| 283 | + |
| 284 | +We align CPU calculation intervals (`-w`) with tmux status bar refresh interval (`status-interval` setting). |
| 285 | + |
| 286 | +### "cmd not ready" message |
| 287 | + |
| 288 | +When you run *tmux* for the first time, you might see following message instead of CPU metric for few seconds. |
| 289 | + |
| 290 | +> cmd ~/.tmux/plugins/sysstat/scripts/cpu.sh not ready |
| 291 | +
|
| 292 | +This is because CPU calculation is long running command, which need to measures CPU average per second for specific time interval. It's not a one-shot command, like memory calculation, which just queries `vmstat` or `proc/meminfo` for current system state. If we did the same for CPU calculation, it would have just print CPU usage average since system bootstrap (not for last X seconds). |
| 293 | + |
| 294 | +Quote from `man tmux`: |
| 295 | +> When constructing formats, tmux does not wait for `#()' commands to finish; instead, the previous result from running the same command is used, or a placeholder if the command has not been run before. If the command hasn't exited, the most recent line of output will be used, but the status line will not be updated more than once a second. |
| 296 | +
|
| 297 | +Therefore, when you run `tmux` for the first time, there is no previous result, thus it prints `<'cmd' not ready>`. Next time status bar is refreshed, tmux will use previous command's output. This is expected behavior. |
| 298 | + |
| 299 | + |
| 300 | +Internals: memory calculation |
| 301 | +---------------------------- |
| 302 | +You might ask what we treat as `free` memory and how it's calculated. |
| 303 | + |
| 304 | +### OSX |
| 305 | +Let's start with OSX. We use `vm_stat` command (not same as `vmstat` on Linux), which reports following data (number of memory pages, not KB): |
| 306 | + |
| 307 | +``` |
| 308 | +$ vm_stat |
| 309 | +Pages free: 37279 |
| 310 | +Pages active: 1514200 |
| 311 | +Pages inactive: 1152997 |
| 312 | +Pages speculative: 6214 |
| 313 | +Pages throttled: 0 |
| 314 | +Pages wired down: 1174408 |
| 315 | +Pages purgeable: 15405 |
| 316 | +Pages stored in compressor: 1615663 |
| 317 | +Pages occupied by compressor: 306717 |
| 318 | +``` |
| 319 | + |
| 320 | +Total installed memory formula is: |
| 321 | +``` |
| 322 | +Total = free + active + inactive + speculative + occupied by compressor + wired |
| 323 | +``` |
| 324 | + |
| 325 | +where |
| 326 | +- `free`, completely unused memory by the system |
| 327 | +- `wired`, critical information stored in RAM by system, kernel and key applications. Never swapped to the hard drive, never replaced with user-level data. |
| 328 | +- `active`, information currently in use or very recently used by applications. When this kind of memory is not used for long (or application is closed), it's move to inactive memory. |
| 329 | +- `inactive`, like buffers/cached memory in Linux. Memory for applications, which recently exited, retained for faster start-up of same application in future. |
| 330 | + |
| 331 | +So the question what constitutes `free` and `used` memory. It turns out, that various monitoring and system statistics tools on OSX each calculate it differently. |
| 332 | + |
| 333 | +- htop: `used = active + wired`, `free` = `total - used` |
| 334 | +- top. Used = `used = active + inactive + occupied by compressor + wired`; Free = `free + speculative` Resident set size (RSS) = `active` |
| 335 | +- OSX activity Monitor. Used = `app memory + wired + compressor`. Note, it's not clear what is app memory. |
| 336 | + |
| 337 | +In general, they either treat currently used memory, which can be reclaimed in case of need (cached, inactive, occupied by compressor), as `used` or `free`. |
| 338 | + |
| 339 | +It makes sense to talk about `available` memory rather than `free` one. Available memory is unused memory + any used memory which can be reclaimed for application needs. |
| 340 | + |
| 341 | +So, `tmux-plugin-sysstat`, uses following formula: |
| 342 | + |
| 343 | +``` |
| 344 | +used = active + wired |
| 345 | +available/free = free/unused + inactive + speculative + occupied by compressor |
| 346 | +``` |
| 347 | + |
| 348 | +### Linux |
| 349 | + |
| 350 | +Same thinking can be applied to Linux systems. |
| 351 | + |
| 352 | +Usually commands like `free` report free/unused, used, buffers, cache memory kinds. |
| 353 | + |
| 354 | +``` |
| 355 | +$ free |
| 356 | + total used free shared buffers cached |
| 357 | +Mem: 1016464 900236 116228 21048 93448 241544 |
| 358 | +-/+ buffers/cache: 565244 451220 |
| 359 | +Swap: 1046524 141712 904812 |
| 360 | +``` |
| 361 | + |
| 362 | +Second line indicates available memory (free + buffers + cache), with an assumption that buffers and cache can be 100% reclaimed in case of need. |
| 363 | + |
| 364 | +However, we're not using free, because its output varies per system. For example on RHEL7, there is no `-/+ buffers/cache`, and `available` memory is reported in different way. We read directly from `/proc/meminfo` |
| 365 | + |
| 366 | +``` |
| 367 | +$ cat /proc/meminfo |
| 368 | +
|
| 369 | +MemTotal: 1016232 kB |
| 370 | +MemFree: 152672 kB |
| 371 | +MemAvailable: 637832 kB |
| 372 | +Buffers: 0 kB |
| 373 | +Cached: 529040 kB |
| 374 | +``` |
| 375 | + |
| 376 | +`tmux-plugin-sysstat` uses following formula: |
| 377 | + |
| 378 | +``` |
| 379 | +free/available = MemAvailable; // if MemAvailable present |
| 380 | +free/available = MemFree + Buffers + Cached; |
| 381 | +used = MemTotal - free/avaialble |
| 382 | +``` |
| 383 | + |
| 384 | +Using `MemAvailable` is more accurate way of getting available memory, rather than manual calculation `free + buffers + cache`, because the assumption that `buffers + cache` can be 100% reclaimed for new application needs might be wrong. When using `MemAvailable`, OS calculates available memory for you, which is apparently better and accurate approach. |
| 385 | + |
| 386 | +See [this topic](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773) on more reasoning about `MemAvailable` field. |
0 commit comments