* Copy in metrics.txt summary of available metrics from mq-golang

* JSON exporter - consistent objecttype for qmgr, merge published and
  polled metrics (#188)
* JSON exporter - allow configuration (`recordmax`) for number of objects
  printed in a single JSON record (also #188)
* AWS exporter -correct interval processing (#192)
* Simplified build/run in a single Dockerfile (#194)

Author: ibmmqmet

CHANGELOG.md

@@ -1,6 +1,15 @@
 # Changelog
 Newest updates are at the top of this file.
 
+### Jan 10 2023 (v5.3.3)
+* Copy in metrics.txt summary of available metrics from mq-golang
+* JSON exporter - consistent objecttype for qmgr, merge published and 
+  polled metrics (#188)
+* JSON exporter - allow configuration (`recordmax`) for number of objects
+  printed in a single JSON record (also #188)
+* AWS exporter -correct interval processing (#192)
+* Simplified build/run in a single Dockerfile (#194)
+
 ### Oct 18 2022 (v5.3.2)
 * Update to MQ 9.3.1
 * Update to use v5.3.2 of mq-golang repository

Dockerfile

@@ -83,17 +83,14 @@ RUN apt-get update \
     ca-certificates \
     && rm -rf /var/lib/apt/lists/*
 
-# Install MQ client 
+# Create MQ client directories
 WORKDIR /opt/mqm
-RUN curl -LO "$RDURL/$VRMF-$RDTAR" \
-    && tar -zxf ./*.tar.gz \
-    && rm -f ./*.tar.gz \
-    && bin/genmqpkg.sh -b /opt/mqm \
-    && mkdir -p /IBM/MQ/data/errors \
+RUN mkdir -p /IBM/MQ/data/errors \
     && mkdir -p /.mqm \
     && chmod -R 777 /IBM \
     && chmod -R 777 /.mqm
 
-COPY --chmod=777 --from=builder /go/bin/${EXPORTER} /opt/bin/${EXPORTER}
+COPY --chmod=555 --from=builder /go/bin/${EXPORTER} /opt/bin/${EXPORTER}
+COPY             --from=builder /opt/mqm/ /opt/mqm/
 
-CMD ["sh", "-c", "/opt/bin/${EXPORTER}"]
+CMD ["sh", "-c", "/opt/bin/${EXPORTER}"]

README.md

@@ -48,7 +48,7 @@ any privileges to install:
 See the README file in the mq-golang repository for more information about any environment variables that may
 be required to point at non-default directories for the MQ C SDK.
 
-### Building a component
+### Building a component on your system directly
 
 * You need to have the MQ client libraries installed first.
 * Create a directory where you want to work with the programs. 
@@ -78,14 +78,24 @@ with getting the agent running in your specific environment.
 The `-mod=vendor` option is important so that the build process does not need to 
 download additional files from external repositories.
 
-## Using a Docker container to build the programs
-You can use the `buildMonitors.sh` script in the `scripts` subdirectory to build a Docker container that
-in turn will build all the binary programs and copy them to a local directory. That script also
-sets some extra version-related flags that will be shown when the program starts. The container will
-automatically download and install the MQ client runtime files needed for compilation.
+## Using containers to build the programs
+The `Dockerfile` in the root directory gives a simple way to both build and run a collector program through
+containers. You still need to provide the configuration file at runtime, perhaps as a mounted volume. For example:
 
-## Building on Windows
-There is a `buildMonitors.bat` file that may help with building on Windows. It assumes you have
+```
+  docker build -t mqprom:1.0 .
+  docker run   -v <directory>/mq_prometheus.yaml:/opt/config/mq_prometheus.yaml mqprom:1.0
+```  
+
+As a more flexible example, you can use the `buildMonitors.sh` script in the `scripts` subdirectory to 
+build a Docker container that in turn will build all the binary programs and copy them to a local directory. 
+That script also sets some extra version-related flags that will be shown when the program starts. The container will
+automatically download and install the MQ client runtime files needed for compilation. This might be a preferred approach when you want to run a collector program alongside 
+a queue manager (perhaps as an MQ SERVICE) and you need to copy the binaries to the
+target system.
+
+## Building to run on Windows
+There is a `buildMonitors.bat` file to help with building on Windows. It assumes you have
 the [tdm-gcc-64](https://jmeubank.github.io/tdm-gcc/download/) 64-bit compiler suite installed. It
 builds all the collectors and corresponding YAML configuration files into %GOPATH%/bin
 
@@ -111,7 +121,7 @@ An alternative collection mechanism uses durable subscriptions for the queue met
 the MAXHANDS attribute on a queue manager. (Queue manager-level metrics are still collected using non-durable subscriptions.)
 
 To set it up, you must provide suitable configuration options. In the
-YAML configuration, these are the attributes (command line or environment variable equivalents exist):
+YAML configuration, these are the attributes (command line or environment variable equivalents also exist):
 - `replyQueue` must refer to a local queue (not a model queue)
 - `replyQueue2` must also be set, referring to a different local queue
 - `durableSubPrefix` is a string that is unique across any collectors that might be connected to this queue manager
@@ -237,7 +247,7 @@ That allows it to be piped from an external stash file or some other
 mechanism. Using the command line flags for controlling passwords is not
 recommended for security-sensitive environments.
 
-Where authentication is needed for access to the database, passwords for those can
+Where authentication is needed for access to a database, passwords for those can
 also be passed via stdin.
 
 ## YAML configuration for all exporters
@@ -272,7 +282,8 @@ The command line flags are highest precedence. Environment variables override se
 
 ## More information
 Each of the sample monitor programs has its own README file describing any particular
-considerations.
+considerations. The metrics.txt file in this directory has a summary of the available
+metrics for each object type.
 
 ## History
 
@@ -289,4 +300,4 @@ in the DCO.
 
 ## Copyright
 
-© Copyright IBM Corporation 2016, 2022
+© Copyright IBM Corporation 2016, 2023

cmd/mq_aws/exporter.go

@@ -1,7 +1,7 @@
 package main
 
 /*
-  Copyright (c) IBM Corporation 2016,2021
+  Copyright (c) IBM Corporation 2016,2022
 
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
@@ -245,23 +245,25 @@ func Collect() error {
 							}
 
 							series = "qmgr"
-							if key != mqmetric.QMgrMapKey {
+							if key == mqmetric.QMgrMapKey {
+								tags["description"] = mqmetric.GetObjectDescription("", ibmmq.MQOT_Q_MGR)
+							} else {
 								tags["queue"] = key
 								series = "queue"
-							}
 
-							usage := ""
-							if usageAttr, ok := mqmetric.GetObjectStatus("", mqmetric.OT_Q).Attributes[mqmetric.ATTR_Q_USAGE].Values[key]; ok {
-								if usageAttr.ValueInt64 == 1 {
-									usage = "XMITQ"
-								} else {
-									usage = "NORMAL"
+								usage := ""
+								if usageAttr, ok := mqmetric.GetObjectStatus("", mqmetric.OT_Q).Attributes[mqmetric.ATTR_Q_USAGE].Values[key]; ok {
+									if usageAttr.ValueInt64 == 1 {
+										usage = "XMITQ"
+									} else {
+										usage = "NORMAL"
+									}
 								}
-							}
 
-							tags["usage"] = usage
-							tags["description"] = mqmetric.GetObjectDescription(key, ibmmq.MQOT_Q)
-							tags["cluster"] = mqmetric.GetQueueAttribute(key, ibmmq.MQCA_CLUSTER_NAME)
+								tags["usage"] = usage
+								tags["description"] = mqmetric.GetObjectDescription(key, ibmmq.MQOT_Q)
+								tags["cluster"] = mqmetric.GetQueueAttribute(key, ibmmq.MQCA_CLUSTER_NAME)
+							}
 
 							pt, _ := newPoint(series+"."+elem.MetricName, t, float64(f), tags)
 							bp.addPoint(pt)
@@ -451,8 +453,9 @@ func Collect() error {
 						qMgrName := strings.TrimSpace(config.cf.QMgrName)
 
 						tags := map[string]string{
-							"qmgr":     qMgrName,
-							"platform": platformString,
+							"qmgr":        qMgrName,
+							"platform":    platformString,
+							"description": mqmetric.GetObjectDescription("", ibmmq.MQOT_Q_MGR),
 						}
 
 						f := mqmetric.QueueManagerNormalise(attr, value.ValueInt64)

cmd/mq_aws/main.go

@@ -49,7 +49,7 @@ func main() {
 
 	if err == nil {
 		interval := config.ci.Interval
-		d, err := time.ParseDuration(interval)
+		d, err = time.ParseDuration(interval)
 		if err != nil || d.Seconds() <= 1 {
 			log.Errorln("Invalid value or too short for interval parameter: ", err)
 			os.Exit(1)

cmd/mq_coll/exporter.go

@@ -200,7 +200,10 @@ func Collect() error {
 							}
 
 							series = "qmgr"
-							if key != mqmetric.QMgrMapKey {
+							if key == mqmetric.QMgrMapKey {
+								tags["description"] = mqmetric.GetObjectDescription("", ibmmq.MQOT_Q_MGR)
+
+							} else {
 								series = "queue"
 								tags[series] = key
 								usage := ""
@@ -365,8 +368,9 @@ func Collect() error {
 					qMgrName := strings.TrimSpace(config.cf.QMgrName)
 
 					tags := map[string]string{
-						"qmgr":     qMgrName,
-						"platform": platformString,
+						"qmgr":        qMgrName,
+						"platform":    platformString,
+						"description": mqmetric.GetObjectDescription("", ibmmq.MQOT_Q_MGR),
 					}
 
 					f := mqmetric.QueueManagerNormalise(attr, value.ValueInt64)

cmd/mq_influx/exporter.go

@@ -230,7 +230,9 @@ func Collect(c client.Client) error {
 							}
 
 							series = "qmgr"
-							if key != mqmetric.QMgrMapKey {
+							if key == mqmetric.QMgrMapKey {
+								tags["description"] = mqmetric.GetObjectDescription("", ibmmq.MQOT_Q_MGR)
+							} else {
 								tags["queue"] = key
 								series = "queue"
 								usage := ""
@@ -422,8 +424,9 @@ func Collect(c client.Client) error {
 							qMgrName := strings.TrimSpace(config.cf.QMgrName)
 
 							tags := map[string]string{
-								"qmgr":     qMgrName,
-								"platform": platformString,
+								"qmgr":        qMgrName,
+								"platform":    platformString,
+								"description": mqmetric.GetObjectDescription("", ibmmq.MQOT_Q_MGR),
 							}
 
 							f := mqmetric.QueueManagerNormalise(attr, value.ValueInt64)

cmd/mq_json/README.md

@@ -96,10 +96,18 @@ jq -c '.collectionTime.timeStamp as $t | .points[] |
      {"time":$t, "chl":.tags.channel, "msg":.metrics.messages}'
 ```
 
-The configuration option `oneline`can be used to print a complete JSON record as a single
+### Collector configuration options
+The collector-specific options can be seen in `config.collector.yaml` in this directory.
+
+* The `oneline` option can be used to print a complete JSON record as a single
 line instead of pretty-printed across mutiple lines. That might be helpful for some tools that
 ingest JSON and prefer to have a single line inputs.
 
+* The `recordmax` option declares how many objects are printed in a single array at each interval. If
+more objects have been recorded, then additional arrays are printed. This may help if processing tools
+have limits on the total record size that can be handled for a single JSON object. Setting the value to `0`
+means there is no effective limit.
+
 ## Metrics
 Once the monitor program has been started, you will see metrics being available.
 More information on the metrics collected through the publish/subscribe

cmd/mq_json/config.collector.yaml

@@ -1,5 +1,10 @@
 
 
 # This is the collector-specific piece of the configuration
+# The defaults here record the output on single lines, with at most 100 objects reported in 
+# each JSON object. Additional objects will be reported in a separate JSON object with the same 
+# timestamp.
 json:
   interval: 10s
+  oneline: true
+  recordmax: 100

cmd/mq_json/config.go

@@ -23,14 +23,16 @@ import (
 )
 
 type mqTTYConfig struct {
-	cf       cf.Config
-	interval string
-	oneline  bool
+	cf        cf.Config
+	interval  string
+	oneline   bool
+	recordmax int
 }
 
 type ConfigYJson struct {
-	Interval string
-	OneLine  bool
+	Interval  string
+	OneLine   bool `yaml:"oneline"`
+	RecordMax int  `yaml:"recordmax"`
 }
 
 type mqExporterConfigYaml struct {
@@ -54,6 +56,7 @@ func initConfig() error {
 
 	cf.AddParm(&config.interval, "10s", cf.CP_STR, "ibmmq.interval", "json", "interval", "How long between each collection")
 	cf.AddParm(&config.oneline, false, cf.CP_BOOL, "ibmmq.oneline", "json", "oneline", "JSON output on a single line")
+	cf.AddParm(&config.recordmax, 100, cf.CP_INT, "ibmmq.recordmax", "json", "recordmax", "Max records in a single JSON array")
 
 	err = cf.ParseParms()
 
@@ -64,6 +67,7 @@ func initConfig() error {
 				cf.CopyYamlConfig(&config.cf, cfy.Global, cfy.Connection, cfy.Objects, cfy.Filters)
 				config.interval = cf.CopyParmIfNotSetStr("json", "interval", cfy.JSON.Interval)
 				config.oneline = cf.CopyParmIfNotSetBool("json", "oneline", cfy.JSON.OneLine)
+				config.recordmax = cf.CopyParmIfNotSetInt("json", "recordmax", cfy.JSON.RecordMax)
 			}
 		}
 	}