Skip to content

Commit 52df03b

Browse files
lizthegreylizthegrey
committed
feat(kafka-receiver): add flags/configs for kafka receiving
1 parent e1186ea commit 52df03b

File tree

12 files changed

+281
-29
lines changed

12 files changed

+281
-29
lines changed

‎config.md‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Honeycomb Refinery Configuration Documentation
22

33
This is the documentation for the configuration file for Honeycomb's Refinery.
4-
It was automatically generated on 2024-02-23 at 22:23:28 UTC.
4+
It was automatically generated on 2024-03-11 at 06:48:05 UTC.
55

66
## The Config file
77

@@ -27,6 +27,7 @@ The remainder of this document describes the sections within the file and the fi
2727
## Table of Contents
2828
- [General Configuration](#general-configuration)
2929
- [Network Configuration](#network-configuration)
30+
- [Kafka Receiver Configuration](#kafka-receiver-configuration)
3031
- [Access Key Configuration](#access-key-configuration)
3132
- [Refinery Telemetry](#refinery-telemetry)
3233
- [Traces](#traces)
@@ -148,6 +149,42 @@ This setting is the destination to which Refinery sends all events that it decid
148149
- Environment variable: `REFINERY_HONEYCOMB_API`
149150
- Command line switch: `--honeycomb-api`
150151

152+
## Kafka Receiver Configuration
153+
154+
`KafkaReceiver` contains configuration options for the Kafka receiver.
155+
### `BootstrapAddr`
156+
157+
BootstrapAddr is the IP and port on which to connect to a Kafka bootstrap server.
158+
159+
This is how we determine the Kafka broker we are using.
160+
161+
- Not eligible for live reload.
162+
- Type: `hostport`
163+
- Environment variable: `REFINERY_KAFKA_BOOTSTRAP_ADDRESS`
164+
- Command line switch: `--kafka-bootstrap-address`
165+
166+
### `Topic`
167+
168+
Topic is the Kafka topic to consume.
169+
170+
This is how we determine the Kafka topic we are using.
171+
172+
- Not eligible for live reload.
173+
- Type: `string`
174+
- Environment variable: `REFINERY_KAFKA_TOPIC`
175+
- Command line switch: `--kafka-topic`
176+
177+
### `ConsumerGroupName`
178+
179+
ConsumerGroupName is the name of the Kafka consumer group to join.
180+
181+
This is how we determine the Kafka consumer group we are using.
182+
183+
- Not eligible for live reload.
184+
- Type: `string`
185+
- Environment variable: `REFINERY_KAFKA_CONSUMER_GROUP_NAME`
186+
- Command line switch: `--kafka-consumer-group-name`
187+
151188
## Access Key Configuration
152189

153190
`AccessKeys` contains access keys -- API keys that the proxy will treat specially, and other flags that control how the proxy handles API keys.

‎config/cmdenv.go‎

Lines changed: 26 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -26,29 +26,32 @@ import (
2626
// that this system uses reflection to establish the relationship between the
2727
// config struct and the command line options.
2828
type CmdEnv struct {
29-
ConfigLocation string `short:"c" long:"config" env:"REFINERY_CONFIG" default:"/etc/refinery/refinery.yaml" description:"config file or URL to load"`
30-
RulesLocation string `short:"r" long:"rules_config" env:"REFINERY_RULES_CONFIG" default:"/etc/refinery/rules.yaml" description:"config file or URL to load"`
31-
HTTPListenAddr string `long:"http-listen-address" env:"REFINERY_HTTP_LISTEN_ADDRESS" description:"HTTP listen address for incoming event traffic"`
32-
PeerListenAddr string `long:"peer-listen-address" env:"REFINERY_PEER_LISTEN_ADDRESS" description:"Peer listen address for communication between Refinery instances"`
33-
GRPCListenAddr string `long:"grpc-listen-address" env:"REFINERY_GRPC_LISTEN_ADDRESS" description:"gRPC listen address for OTLP traffic"`
34-
RedisHost string `long:"redis-host" env:"REFINERY_REDIS_HOST" description:"Redis host address"`
35-
RedisUsername string `long:"redis-username" env:"REFINERY_REDIS_USERNAME" description:"Redis username"`
36-
RedisPassword string `long:"redis-password" env:"REFINERY_REDIS_PASSWORD" description:"Redis password"`
37-
RedisAuthCode string `long:"redis-auth-code" env:"REFINERY_REDIS_AUTH_CODE" description:"Redis AUTH code"`
38-
HoneycombAPI string `long:"honeycomb-api" env:"REFINERY_HONEYCOMB_API" description:"Honeycomb API URL"`
39-
HoneycombAPIKey string `long:"honeycomb-api-key" env:"REFINERY_HONEYCOMB_API_KEY" description:"Honeycomb API key (for logger and metrics)"`
40-
HoneycombLoggerAPIKey string `long:"logger-api-key" env:"REFINERY_HONEYCOMB_LOGGER_API_KEY" description:"Honeycomb logger API key"`
41-
LegacyMetricsAPIKey string `long:"legacy-metrics-api-key" env:"REFINERY_HONEYCOMB_METRICS_API_KEY" description:"API key for legacy Honeycomb metrics"`
42-
OTelMetricsAPIKey string `long:"otel-metrics-api-key" env:"REFINERY_OTEL_METRICS_API_KEY" description:"API key for OTel metrics if being sent to Honeycomb"`
43-
QueryAuthToken string `long:"query-auth-token" env:"REFINERY_QUERY_AUTH_TOKEN" description:"Token for debug/management queries"`
44-
AvailableMemory MemorySize `long:"available-memory" env:"REFINERY_AVAILABLE_MEMORY" description:"The maximum memory available for Refinery to use (ex: 4GiB)."`
45-
Debug bool `short:"d" long:"debug" description:"Runs debug service (on the first open port between localhost:6060 and :6069 by default)"`
46-
Version bool `short:"v" long:"version" description:"Print version number and exit"`
47-
InterfaceNames bool `long:"interface-names" description:"Print system's network interface names and exit."`
48-
Validate bool `short:"V" long:"validate" description:"Validate the configuration files, writing results to stdout, and exit with 0 if valid, 1 if invalid."`
49-
NoValidate bool `long:"no-validate" description:"Do not attempt to validate the configuration files. Makes --validate meaningless."`
50-
WriteConfig string `long:"write-config" description:"After applying defaults, environment variables, and command line values, write the loaded configuration to the specified file as YAML and exit."`
51-
WriteRules string `long:"write-rules" description:"After applying defaults, write the loaded rules to the specified file as YAML and exit."`
29+
ConfigLocation string `short:"c" long:"config" env:"REFINERY_CONFIG" default:"/etc/refinery/refinery.yaml" description:"config file or URL to load"`
30+
RulesLocation string `short:"r" long:"rules_config" env:"REFINERY_RULES_CONFIG" default:"/etc/refinery/rules.yaml" description:"config file or URL to load"`
31+
HTTPListenAddr string `long:"http-listen-address" env:"REFINERY_HTTP_LISTEN_ADDRESS" description:"HTTP listen address for incoming event traffic"`
32+
PeerListenAddr string `long:"peer-listen-address" env:"REFINERY_PEER_LISTEN_ADDRESS" description:"Peer listen address for communication between Refinery instances"`
33+
GRPCListenAddr string `long:"grpc-listen-address" env:"REFINERY_GRPC_LISTEN_ADDRESS" description:"gRPC listen address for OTLP traffic"`
34+
KafkaBootstrapAddr string `long:"kafka-bootstrap-address" env:"REFINERY_KAFKA_BOOTSTRAP_ADDRESS" description:"Kafka bootstrap address"`
35+
KafkaTopic string `long:"kafka-topic" env:"REFINERY_KAFKA_TOPIC" description:"Kafka topic to consume"`
36+
KafkaConsumerGroupName string `long:"kafka-consumer-group-name" env:"REFINERY_KAFKA_CONSUMER_GROUP_NAME" description:"Kafka consumer group to join"`
37+
RedisHost string `long:"redis-host" env:"REFINERY_REDIS_HOST" description:"Redis host address"`
38+
RedisUsername string `long:"redis-username" env:"REFINERY_REDIS_USERNAME" description:"Redis username"`
39+
RedisPassword string `long:"redis-password" env:"REFINERY_REDIS_PASSWORD" description:"Redis password"`
40+
RedisAuthCode string `long:"redis-auth-code" env:"REFINERY_REDIS_AUTH_CODE" description:"Redis AUTH code"`
41+
HoneycombAPI string `long:"honeycomb-api" env:"REFINERY_HONEYCOMB_API" description:"Honeycomb API URL"`
42+
HoneycombAPIKey string `long:"honeycomb-api-key" env:"REFINERY_HONEYCOMB_API_KEY" description:"Honeycomb API key (for logger and metrics)"`
43+
HoneycombLoggerAPIKey string `long:"logger-api-key" env:"REFINERY_HONEYCOMB_LOGGER_API_KEY" description:"Honeycomb logger API key"`
44+
LegacyMetricsAPIKey string `long:"legacy-metrics-api-key" env:"REFINERY_HONEYCOMB_METRICS_API_KEY" description:"API key for legacy Honeycomb metrics"`
45+
OTelMetricsAPIKey string `long:"otel-metrics-api-key" env:"REFINERY_OTEL_METRICS_API_KEY" description:"API key for OTel metrics if being sent to Honeycomb"`
46+
QueryAuthToken string `long:"query-auth-token" env:"REFINERY_QUERY_AUTH_TOKEN" description:"Token for debug/management queries"`
47+
AvailableMemory MemorySize `long:"available-memory" env:"REFINERY_AVAILABLE_MEMORY" description:"The maximum memory available for Refinery to use (ex: 4GiB)."`
48+
Debug bool `short:"d" long:"debug" description:"Runs debug service (on the first open port between localhost:6060 and :6069 by default)"`
49+
Version bool `short:"v" long:"version" description:"Print version number and exit"`
50+
InterfaceNames bool `long:"interface-names" description:"Print system's network interface names and exit."`
51+
Validate bool `short:"V" long:"validate" description:"Validate the configuration files, writing results to stdout, and exit with 0 if valid, 1 if invalid."`
52+
NoValidate bool `long:"no-validate" description:"Do not attempt to validate the configuration files. Makes --validate meaningless."`
53+
WriteConfig string `long:"write-config" description:"After applying defaults, environment variables, and command line values, write the loaded configuration to the specified file as YAML and exit."`
54+
WriteRules string `long:"write-rules" description:"After applying defaults, write the loaded rules to the specified file as YAML and exit."`
5255
}
5356

5457
func NewCmdEnvOptions(args []string) (*CmdEnv, error) {

‎config/config.go‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,16 @@ type Config interface {
3434
// GetHTTPIdleTimeout returns the idle timeout for refinery's HTTP server
3535
GetHTTPIdleTimeout() time.Duration
3636

37+
// GetKafkaBootstrapAddr returns the address and port on which to connect
38+
// to become a Kafka consumer.
39+
GetKafkaBootstrapAddr() (string, error)
40+
41+
// GetKafkaTopic returns the topic to consume.
42+
GetKafkaTopic() (string, error)
43+
44+
// GetKafkaConsumerGroupsName returns the Kafka consumer group to join.
45+
GetKafkaConsumerGroupName() (string, error)
46+
3747
// GetCompressPeerCommunication will be true if refinery should compress
3848
// data before forwarding it to a peer.
3949
GetCompressPeerCommunication() bool

‎config/file_config.go‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ type fileConfig struct {
4949
type configContents struct {
5050
General GeneralConfig `yaml:"General"`
5151
Network NetworkConfig `yaml:"Network"`
52+
KafkaReceiver KafkaReceiverConfig `yaml:"KafkaReceiver"`
5253
AccessKeys AccessKeyConfig `yaml:"AccessKeys"`
5354
Telemetry RefineryTelemetryConfig `yaml:"RefineryTelemetry"`
5455
Traces TracesConfig `yaml:"Traces"`
@@ -84,6 +85,12 @@ type NetworkConfig struct {
8485
HTTPIdleTimeout Duration `yaml:"HTTPIdleTimeout"`
8586
}
8687

88+
type KafkaReceiverConfig struct {
89+
BootstrapAddr string `yaml:"BootstrapAddr" cmdenv:"KafkaBootstrapAddr"`
90+
Topic string `yaml:"Topic" cmdenv:"KafkaTopic"`
91+
ConsumerGroupName string `yaml:"ConsumerGroupName" cmdenv:"KafkaConsumerGroupName"`
92+
}
93+
8794
type AccessKeyConfig struct {
8895
ReceiveKeys []string `yaml:"ReceiveKeys" default:"[]"`
8996
AcceptOnlyListedKeys bool `yaml:"AcceptOnlyListedKeys"`
@@ -505,6 +512,32 @@ func (f *fileConfig) GetHTTPIdleTimeout() time.Duration {
505512
return time.Duration(f.mainConfig.Network.HTTPIdleTimeout)
506513
}
507514

515+
func (f *fileConfig) GetKafkaBootstrapAddr() (string, error) {
516+
f.mux.RLock()
517+
defer f.mux.RUnlock()
518+
519+
addr := f.mainConfig.KafkaReceiver.BootstrapAddr
520+
_, _, err := net.SplitHostPort(addr)
521+
if addr != "" && err != nil {
522+
return "", err
523+
}
524+
return f.mainConfig.KafkaReceiver.BootstrapAddr, nil
525+
}
526+
527+
func (f *fileConfig) GetKafkaTopic() (string, error) {
528+
f.mux.RLock()
529+
defer f.mux.RUnlock()
530+
531+
return f.mainConfig.KafkaReceiver.Topic, nil
532+
}
533+
534+
func (f *fileConfig) GetKafkaConsumerGroupName() (string, error) {
535+
f.mux.RLock()
536+
defer f.mux.RUnlock()
537+
538+
return f.mainConfig.KafkaReceiver.ConsumerGroupName, nil
539+
}
540+
508541
func (f *fileConfig) GetCompressPeerCommunication() bool {
509542
f.mux.RLock()
510543
defer f.mux.RUnlock()

‎config/metadata/configMeta.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,43 @@ groups:
140140
This setting is the destination to which Refinery sends all events
141141
that it decides to keep.
142142
143+
- name: KafkaReceiver
144+
title: "Kafka Receiver Configuration"
145+
description: contains configuration options for the Kafka receiver.
146+
fields:
147+
- name: BootstrapAddr
148+
type: hostport
149+
valuetype: nondefault
150+
reload: false
151+
default: ""
152+
envvar: REFINERY_KAFKA_BOOTSTRAP_ADDRESS
153+
commandLine: kafka-bootstrap-address
154+
summary: is the IP and port on which to connect to a Kafka bootstrap server.
155+
description: >
156+
This is how we determine the Kafka broker we are using.
157+
158+
- name: Topic
159+
type: string
160+
valuetype: nondefault
161+
reload: false
162+
default: ""
163+
envvar: REFINERY_KAFKA_TOPIC
164+
commandLine: kafka-topic
165+
summary: is the Kafka topic to consume.
166+
description: >
167+
This is how we determine the Kafka topic we are using.
168+
169+
- name: ConsumerGroupName
170+
type: string
171+
valuetype: nondefault
172+
reload: false
173+
default: ""
174+
envvar: REFINERY_KAFKA_CONSUMER_GROUP_NAME
175+
commandLine: kafka-consumer-group-name
176+
summary: is the name of the Kafka consumer group to join.
177+
description: >
178+
This is how we determine the Kafka consumer group we are using.
179+
143180
- name: AccessKeys
144181
title: "Access Key Configuration"
145182
description: >

‎config/mock.go‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,10 @@ type MockConfig struct {
9191
ParentIdFieldNames []string
9292
CfgMetadata []ConfigMetadata
9393

94+
KafkaBootstrapAddr string
95+
KafkaTopic string
96+
KafkaConsumerGroupName string
97+
9498
Mux sync.RWMutex
9599
}
96100

@@ -163,6 +167,27 @@ func (m *MockConfig) GetHTTPIdleTimeout() time.Duration {
163167
return m.GetHTTPIdleTimeoutVal
164168
}
165169

170+
func (m *MockConfig) GetKafkaBootstrapAddr() (string, error) {
171+
m.Mux.RLock()
172+
defer m.Mux.RUnlock()
173+
174+
return m.KafkaBootstrapAddr, nil
175+
}
176+
177+
func (m *MockConfig) GetKafkaTopic() (string, error) {
178+
m.Mux.RLock()
179+
defer m.Mux.RUnlock()
180+
181+
return m.KafkaTopic, nil
182+
}
183+
184+
func (m *MockConfig) GetKafkaConsumerGroupName() (string, error) {
185+
m.Mux.RLock()
186+
defer m.Mux.RUnlock()
187+
188+
return m.KafkaConsumerGroupName, nil
189+
}
190+
166191
func (m *MockConfig) GetCompressPeerCommunication() bool {
167192
m.Mux.RLock()
168193
defer m.Mux.RUnlock()

‎config_complete.yaml‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
## Honeycomb Refinery Configuration ##
33
######################################
44
#
5-
# created on 2024-02-23 at 22:23:27 UTC from ../../config.yaml using a template generated on 2024-02-23 at 22:23:25 UTC
5+
# created on 2024-03-11 at 06:48:04 UTC from ../../config.yaml using a template generated on 2024-03-11 at 06:48:03 UTC
66

77
# This file contains a configuration for the Honeycomb Refinery. It is in YAML
88
# format, organized into named groups, each of which contains a set of
@@ -125,6 +125,35 @@ Network:
125125
## Eligible for live reload.
126126
# HoneycombAPI: "https://api.honeycomb.io"
127127

128+
##################################
129+
## Kafka Receiver Configuration ##
130+
##################################
131+
KafkaReceiver:
132+
## KafkaReceiver contains configuration options for the Kafka receiver.
133+
####
134+
## BootstrapAddr is the IP and port on which to connect to a Kafka
135+
## bootstrap server.
136+
##
137+
## This is how we determine the Kafka broker we are using.
138+
##
139+
## Should be an ip:port like "".
140+
## Not eligible for live reload.
141+
# BootstrapAddr: ""
142+
143+
## Topic is the Kafka topic to consume.
144+
##
145+
## This is how we determine the Kafka topic we are using.
146+
##
147+
## Not eligible for live reload.
148+
# Topic: ""
149+
150+
## ConsumerGroupName is the name of the Kafka consumer group to join.
151+
##
152+
## This is how we determine the Kafka consumer group we are using.
153+
##
154+
## Not eligible for live reload.
155+
# ConsumerGroupName: ""
156+
128157
##############################
129158
## Access Key Configuration ##
130159
##############################

‎refinery_config.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,43 @@ This setting is the destination to which Refinery sends all events that it decid
125125
- Environment variable: `REFINERY_HONEYCOMB_API`
126126
- Command line switch: `--honeycomb-api`
127127

128+
## Kafka Receiver Configuration
129+
130+
`KafkaReceiver` contains configuration options for the Kafka receiver.
131+
132+
### `BootstrapAddr`
133+
134+
`BootstrapAddr` is the IP and port on which to connect to a Kafka bootstrap server.
135+
136+
This is how we determine the Kafka broker we are using.
137+
138+
- Not eligible for live reload.
139+
- Type: `hostport`
140+
- Environment variable: `REFINERY_KAFKA_BOOTSTRAP_ADDRESS`
141+
- Command line switch: `--kafka-bootstrap-address`
142+
143+
### `Topic`
144+
145+
`Topic` is the Kafka topic to consume.
146+
147+
This is how we determine the Kafka topic we are using.
148+
149+
- Not eligible for live reload.
150+
- Type: `string`
151+
- Environment variable: `REFINERY_KAFKA_TOPIC`
152+
- Command line switch: `--kafka-topic`
153+
154+
### `ConsumerGroupName`
155+
156+
`ConsumerGroupName` is the name of the Kafka consumer group to join.
157+
158+
This is how we determine the Kafka consumer group we are using.
159+
160+
- Not eligible for live reload.
161+
- Type: `string`
162+
- Environment variable: `REFINERY_KAFKA_CONSUMER_GROUP_NAME`
163+
- Command line switch: `--kafka-consumer-group-name`
164+
128165
## Access Key Configuration
129166

130167
`AccessKeys` contains access keys -- API keys that the proxy will treat specially, and other flags that control how the proxy handles API keys.

‎rules.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Honeycomb Refinery Rules Documentation
22

33
This is the documentation for the rules configuration for Honeycomb's Refinery.
4-
It was automatically generated on 2024-02-23 at 22:23:29 UTC.
4+
It was automatically generated on 2024-03-11 at 06:48:05 UTC.
55

66
## The Rules file
77

‎tools/convert/configDataNames.txt‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Names of groups and fields in the new config file format.
2-
# Automatically generated on 2024-02-23 at 22:23:26 UTC.
2+
# Automatically generated on 2024-03-11 at 06:48:03 UTC.
33

44
General:
55
- ConfigurationVersion
@@ -21,6 +21,14 @@ Network:
2121
- HoneycombAPI
2222

2323

24+
KafkaReceiver:
25+
- BootstrapAddr
26+
27+
- Topic
28+
29+
- ConsumerGroupName
30+
31+
2432
AccessKeys:
2533
- ReceiveKeys (originally APIKeys)
2634

0 commit comments

Comments
 (0)