Skip to content

Commit fb465ac

Browse files
ag-ramachandranag-ramachandran
committed
* Set up dependencies explicitly
1 parent 32d0021 commit fb465ac

File tree

3 files changed

+223
-21
lines changed

3 files changed

+223
-21
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Copilot Instructions
2+
3+
## Project Overview
4+
5+
This is a Logstash output plugin (`logstash-output-kusto`) that sends events from Logstash to Azure Data Explorer (Kusto). It is a JRuby gem that wraps the Azure Kusto Java SDK. The plugin runs on the JVM via Logstash's embedded JRuby runtime.
6+
7+
## Architecture
8+
9+
The plugin has three core classes, all nested under `LogStash::Outputs::Kusto`:
10+
11+
- **`Kusto` (kusto.rb)** — Main output plugin. Extends `LogStash::Outputs::Base`. Handles Logstash config registration, file I/O (writing events to temp files with time-based rotation), and lifecycle (`register`, `multi_receive_encoded`, `close`). Uses `concurrency :shared`.
12+
- **`Ingestor` (kusto/ingestor.rb)** — Manages authentication (AAD app credentials, managed identity, or CLI auth) and async upload to Kusto via the Java SDK. Runs uploads on a `Concurrent::ThreadPoolExecutor`. Retries indefinitely on transient upload failures.
13+
- **`Interval` (kusto/interval.rb)** — Simple timer utility that runs a callable at a fixed interval using a background thread with `Mutex`/`ConditionVariable`.
14+
- **`IOWriter`** — Wrapper around file descriptors at the bottom of `kusto.rb`, tracking active/inactive state for stale file cleanup.
15+
16+
Data flow: Events → codec encodes to JSON lines → written to time-rotated temp files → stale files closed → `Ingestor.upload_async` queues file for Kusto ingestion via Java SDK → temp file deleted on success.
17+
18+
## Build System
19+
20+
This project uses a dual build system — Gradle for Java dependency management and Bundler/Gem for JRuby packaging.
21+
22+
**Vendor Java dependencies (required before tests or gem build):**
23+
```sh
24+
./gradlew vendor
25+
```
26+
This resolves the Kusto Java SDK and all transitive dependencies, copies JARs to `vendor/jar-dependencies/`, and generates `lib/logstash-output-kusto_jars.rb`.
27+
28+
**Install Ruby dependencies:**
29+
```sh
30+
jruby -S bundle install
31+
```
32+
33+
**Run all unit tests:**
34+
```sh
35+
jruby -S bundle exec rspec
36+
```
37+
38+
**Run a single test file:**
39+
```sh
40+
jruby -S bundle exec rspec spec/outputs/kusto_spec.rb
41+
```
42+
43+
**Run a single test by line number:**
44+
```sh
45+
jruby -S bundle exec rspec spec/outputs/kusto/ingestor_spec.rb:25
46+
```
47+
48+
**Build the gem:**
49+
```sh
50+
jruby -S gem build logstash-output-kusto.gemspec
51+
```
52+
53+
**Full build sequence (from scratch):**
54+
```sh
55+
bundle install && ./gradlew vendor && jruby -S bundle exec rspec && jruby -S gem build *.gemspec
56+
```
57+
58+
## Running Locally
59+
60+
See `run.sh` for the full local workflow. It requires a local Logstash installation and the following environment variables:
61+
62+
```sh
63+
export LOGSTASH_SOURCE=1
64+
export LOGSTASH_PATH="/path/to/logstash" # local Logstash installation root
65+
export JRUBY_HOME=$LOGSTASH_PATH/vendor/jruby
66+
export JAVA_HOME=$LOGSTASH_PATH/jdk
67+
export PATH=$JRUBY_HOME/bin:$JAVA_HOME/bin:$LOGSTASH_PATH/bin/logstash:$PATH
68+
```
69+
70+
The script runs the full pipeline: install bundler → bundle install → vendor JARs → run unit tests → build gem → install the gem into local Logstash → run e2e tests. E2e tests also require Kusto cluster connection variables (`ENGINE_URL`, `INGEST_URL`, `TEST_DATABASE`) and use Azure CLI auth.
71+
72+
## Linting
73+
74+
RuboCop is configured (`.rubocop.yml`) with max line length of 120 characters.
75+
76+
## Key Conventions
77+
78+
- **JRuby required** — All Ruby commands must use `jruby -S` (not `ruby`). The gem platform is `java` and the code uses Java interop extensively.
79+
- **Version source of truth** — The gem version is read from the `version` file in the project root (not hardcoded in the gemspec).
80+
- **Java interop pattern** — Java classes are accessed via `Java::com.microsoft.azure.kusto.*` namespace. The `_jars.rb` file is auto-generated by Gradle — never edit it manually.
81+
- **Logstash plugin conventions** — Config parameters use `config :name, validate: :type, default: value`. The plugin uses `config_name 'kusto'` and `default :codec, 'json_lines'`.
82+
- **Authentication modes** — Three mutually exclusive auth paths: AAD app credentials (`app_id`/`app_key`/`app_tenant`), managed identity (`managed_identity`), or Azure CLI (`cli_auth` — dev/test only).
83+
- **Dynamic field references** — Logstash `%{field}` references are allowed in `path` (for time-based file rotation) but explicitly forbidden in `database`, `table`, and `json_mapping`.
84+
- **Test style** — RSpec with `logstash-devutils`. Tests mock the logger with `spy('logger')`. The `spec_helpers.rb` captures stdout/stderr in an `around` filter and sets log level to debug.

‎build.gradle‎

Lines changed: 138 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -42,19 +42,107 @@ repositories {
4242
// update dependencies to bom azure-sdk-bom/1.2.37
4343

4444
dependencies {
45-
// Kusto client libraries (updated)
46-
implementation "com.microsoft.azure.kusto:kusto-data:${kustoVersion}"
47-
implementation "com.microsoft.azure.kusto:kusto-ingest:${kustoVersion}"
45+
// Kusto client libraries — all transitive deps that we explicitly pin are excluded
46+
// to prevent older versions from being pulled in
47+
implementation("com.microsoft.azure.kusto:kusto-data:${kustoVersion}") {
48+
exclude group: 'com.azure'
49+
exclude group: 'com.fasterxml.jackson.core'
50+
exclude group: 'com.fasterxml.jackson.datatype'
51+
exclude group: 'com.fasterxml.woodstox'
52+
exclude group: 'com.microsoft.azure'
53+
exclude group: 'commons-codec'
54+
exclude group: 'commons-logging'
55+
exclude group: 'io.netty'
56+
exclude group: 'io.projectreactor'
57+
exclude group: 'io.vavr'
58+
exclude group: 'net.java.dev.jna'
59+
exclude group: 'org.apache.httpcomponents'
60+
exclude group: 'org.codehaus.woodstox'
61+
exclude group: 'org.ow2.asm'
62+
exclude group: 'org.reactivestreams'
63+
exclude group: 'org.slf4j'
64+
}
65+
implementation("com.microsoft.azure.kusto:kusto-ingest:${kustoVersion}") {
66+
exclude group: 'com.azure'
67+
exclude group: 'com.fasterxml.jackson.core'
68+
exclude group: 'com.fasterxml.jackson.datatype'
69+
exclude group: 'com.fasterxml.woodstox'
70+
exclude group: 'com.microsoft.azure'
71+
exclude group: 'commons-codec'
72+
exclude group: 'commons-logging'
73+
exclude group: 'io.netty'
74+
exclude group: 'io.projectreactor'
75+
exclude group: 'io.vavr'
76+
exclude group: 'net.java.dev.jna'
77+
exclude group: 'org.apache.httpcomponents'
78+
exclude group: 'org.codehaus.woodstox'
79+
exclude group: 'org.ow2.asm'
80+
exclude group: 'org.reactivestreams'
81+
exclude group: 'org.slf4j'
82+
}
4883

49-
// Azure client libraries (versions will be resolved by the BOM)
50-
implementation 'com.azure:azure-core-http-netty:1.16.3'
51-
implementation 'com.azure:azure-core:1.57.1'
52-
implementation 'com.azure:azure-data-tables:12.5.9'
53-
implementation 'com.azure:azure-identity:1.18.2'
84+
// Azure client libraries
85+
implementation('com.azure:azure-core-http-netty:1.16.3') {
86+
exclude group: 'io.netty'
87+
exclude group: 'io.projectreactor.netty'
88+
exclude group: 'com.fasterxml.jackson.core'
89+
exclude group: 'com.fasterxml.jackson.datatype'
90+
exclude group: 'org.slf4j'
91+
exclude group: 'io.projectreactor'
92+
}
93+
implementation('com.azure:azure-core:1.57.1') {
94+
exclude group: 'com.fasterxml.jackson.core'
95+
exclude group: 'com.fasterxml.jackson.datatype'
96+
exclude group: 'com.fasterxml.woodstox'
97+
exclude group: 'org.codehaus.woodstox'
98+
exclude group: 'org.slf4j'
99+
exclude group: 'io.projectreactor'
100+
}
101+
implementation('com.azure:azure-data-tables:12.5.9') {
102+
exclude group: 'io.netty'
103+
exclude group: 'io.projectreactor.netty'
104+
exclude group: 'com.fasterxml.jackson.core'
105+
exclude group: 'com.fasterxml.jackson.datatype'
106+
exclude group: 'org.slf4j'
107+
exclude group: 'io.projectreactor'
108+
}
109+
implementation('com.azure:azure-identity:1.18.2') {
110+
exclude group: 'com.microsoft.azure', module: 'msal4j'
111+
exclude group: 'io.netty'
112+
exclude group: 'io.projectreactor.netty'
113+
exclude group: 'com.fasterxml.jackson.core'
114+
exclude group: 'com.fasterxml.jackson.datatype'
115+
exclude group: 'net.java.dev.jna'
116+
exclude group: 'org.ow2.asm'
117+
exclude group: 'org.slf4j'
118+
exclude group: 'io.projectreactor'
119+
}
54120
implementation 'com.azure:azure-json:1.5.1'
55-
implementation 'com.azure:azure-storage-blob:12.33.2'
56-
implementation 'com.azure:azure-storage-common:12.32.2'
57-
implementation 'com.azure:azure-storage-queue:12.28.2'
121+
implementation('com.azure:azure-storage-blob:12.33.2') {
122+
exclude group: 'io.netty'
123+
exclude group: 'io.projectreactor.netty'
124+
exclude group: 'com.fasterxml.jackson.core'
125+
exclude group: 'com.fasterxml.jackson.datatype'
126+
exclude group: 'org.slf4j'
127+
exclude group: 'io.projectreactor'
128+
exclude module: 'azure-storage-common'
129+
}
130+
implementation('com.azure:azure-storage-common:12.32.2') {
131+
exclude group: 'io.netty'
132+
exclude group: 'io.projectreactor.netty'
133+
exclude group: 'com.fasterxml.jackson.core'
134+
exclude group: 'com.fasterxml.jackson.datatype'
135+
exclude group: 'org.slf4j'
136+
exclude group: 'io.projectreactor'
137+
}
138+
implementation('com.azure:azure-storage-queue:12.28.2') {
139+
exclude group: 'io.netty'
140+
exclude group: 'io.projectreactor.netty'
141+
exclude group: 'com.fasterxml.jackson.core'
142+
exclude group: 'com.fasterxml.jackson.datatype'
143+
exclude group: 'org.slf4j'
144+
exclude group: 'io.projectreactor'
145+
}
58146
implementation 'com.azure:azure-xml:1.2.1'
59147

60148
// Jackson - bump to a newer 2.19.x patch
@@ -66,16 +154,28 @@ dependencies {
66154
implementation 'com.fasterxml.woodstox:woodstox-core:7.1.1'
67155
implementation 'com.github.stephenc.jcip:jcip-annotations:1.0-1'
68156
// MSAL4J bump
69-
implementation 'com.microsoft.azure:msal4j:1.23.1'
157+
implementation('com.microsoft.azure:msal4j:1.23.1') {
158+
exclude group: 'com.azure'
159+
exclude group: 'net.java.dev.jna'
160+
exclude group: 'org.slf4j'
161+
}
70162
implementation 'com.nimbusds:content-type:2.3'
71163
implementation 'com.nimbusds:lang-tag:1.7'
72164
implementation 'com.nimbusds:nimbus-jose-jwt:10.6'
73-
implementation 'com.nimbusds:oauth2-oidc-sdk:11.31'
165+
implementation('com.nimbusds:oauth2-oidc-sdk:11.31') {
166+
exclude group: 'org.ow2.asm'
167+
}
74168
implementation 'com.univocity:univocity-parsers:2.9.1'
75169
implementation 'commons-codec:commons-codec:1.19.0'
76170
implementation 'commons-logging:commons-logging:1.3.1'
77-
implementation "io.github.resilience4j:resilience4j-core:${resilience4jVersion}"
78-
implementation "io.github.resilience4j:resilience4j-retry:${resilience4jVersion}"
171+
implementation("io.github.resilience4j:resilience4j-core:${resilience4jVersion}") {
172+
exclude group: 'io.vavr'
173+
exclude group: 'org.slf4j'
174+
}
175+
implementation("io.github.resilience4j:resilience4j-retry:${resilience4jVersion}") {
176+
exclude group: 'io.vavr'
177+
exclude group: 'org.slf4j'
178+
}
79179
implementation "io.netty:netty-buffer:${nettyVersion}"
80180
implementation "io.netty:netty-codec-dns:${nettyVersion}"
81181
implementation "io.netty:netty-codec-http2:${nettyVersion}"
@@ -97,17 +197,30 @@ dependencies {
97197
implementation "io.netty:netty-transport-native-kqueue:${nettyVersion}:osx-x86_64"
98198
implementation "io.netty:netty-transport-native-unix-common:${nettyVersion}"
99199
implementation "io.netty:netty-transport:${nettyVersion}"
100-
implementation "io.projectreactor.netty:reactor-netty-core:${reactorNettyVersion}"
101-
implementation "io.projectreactor.netty:reactor-netty-http:${reactorNettyVersion}"
200+
implementation("io.projectreactor.netty:reactor-netty-core:${reactorNettyVersion}") {
201+
exclude group: 'io.netty'
202+
exclude group: 'io.projectreactor'
203+
}
204+
implementation("io.projectreactor.netty:reactor-netty-http:${reactorNettyVersion}") {
205+
exclude group: 'io.netty'
206+
exclude group: 'io.projectreactor'
207+
}
102208
implementation "io.projectreactor:reactor-core:3.8.1"
103209
implementation "io.vavr:vavr:${vavrVersion}"
104210
implementation "io.vavr:vavr-match:${vavrVersion}"
105211
implementation "net.java.dev.jna:jna-platform:${jnaVersion}"
106212
implementation "net.java.dev.jna:jna:${jnaVersion}"
107-
implementation "net.minidev:accessors-smart:${minidevVersion}"
108-
implementation "net.minidev:json-smart:${minidevVersion}"
213+
implementation("net.minidev:accessors-smart:${minidevVersion}") {
214+
exclude group: 'org.ow2.asm'
215+
}
216+
implementation("net.minidev:json-smart:${minidevVersion}") {
217+
exclude group: 'org.ow2.asm'
218+
}
109219
implementation 'org.apache.commons:commons-text:1.15.0'
110-
implementation 'org.apache.httpcomponents:httpclient:4.5.14'
220+
implementation('org.apache.httpcomponents:httpclient:4.5.14') {
221+
exclude group: 'commons-codec'
222+
exclude group: 'commons-logging'
223+
}
111224
implementation 'org.apache.httpcomponents:httpcore:4.4.16'
112225
implementation 'org.codehaus.woodstox:stax2-api:4.2.2'
113226
implementation 'org.jetbrains:annotations:24.1.0'
@@ -146,6 +259,11 @@ task vendor {
146259
// take in all the dependencies from the runtimeClasspath and copy them to the vendor/jar-dependencies folder
147260
doLast {
148261
String vendorPathPrefix = "vendor/jar-dependencies"
262+
// Clean old vendor JARs to prevent stale versions from accumulating
263+
File vendorDir = file(vendorPathPrefix)
264+
if (vendorDir.exists()) {
265+
vendorDir.deleteDir()
266+
}
149267
configurations.runtimeClasspath.resolvedConfiguration.resolvedArtifacts.each { artifact ->
150268
def dep = artifact.moduleVersion.id
151269
println("Copying ${dep.group}:${dep.name}:${dep.version}")

‎version‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
2.1.5
1+
2.1.6

0 commit comments

Comments
 (0)