Proxy support #304

mashhurs · 2025-06-06T00:22:53Z

Description

Introduces proxy config to accept proxy URI to connect to Elasticsearch.

Closes [Feature]: Add proxy support for the Elastic Integration Filter #303

Test

For the local test, used SquidMan
Config

// generator
...
filter {
    elastic_integration {
        # cloud
        id => "es_integ_1"
        cloud_id => "my-cloud-id"
        api_key => "api-key"
        proxy => "http://127.0.0.1:8083"
    }

// output

When proxy is stopped, cannot connect to ES

[2025-06-05T17:06:16,068][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Fetching Elasticsearch version information failed: Connection refused>, :backtrace=>["/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:306:in `raise_config_error!'", "/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:395:in `connected_es_version_info'", "/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:428:in `serverless?'", "/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:351:in `initialize_elasticsearch_rest_client!'", "/Users/mashhur/Dev/elastic/ls-plugins/logstash-filter-elastic_integration/lib/logstash/filters/elastic_integration.rb:136:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in `block in register_plugins'", "org/jruby/RubyArray.java:1981:in `each'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `register_plugins'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:623:in `maybe_setup_out_plugins'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:257:in `start_workers'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:198:in `run'", "/Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:150:in `block in start'"], "pipeline.sources"=>["/Users/mashhur/Dev/elastic/logstash/config/elastic_integration_simple.conf"], :thread=>"#<Thread:0x5bda8081 /Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}

When connected, successfully processes the data stream pipeline executions

╰─➤  bin/logstash -f config/elastic_integration_simple.conf --enable-local-plugin-development
Using system java: /Users/mashhur/.sdkman/candidates/java/current/bin/java
Sending Logstash logs to /Users/mashhur/Dev/elastic/logstash/logs which is now configured via log4j2.properties
[2025-06-05T17:14:24,266][INFO ][logstash.runner          ] Log4j configuration path used is: /Users/mashhur/Dev/elastic/logstash/config/log4j2.properties
[2025-06-05T17:14:24,270][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2025-06-05T17:14:24,271][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"9.0.2", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +indy +jit [arm64-darwin]"}
[2025-06-05T17:14:24,272][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2025-06-05T17:14:24,293][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000` (logstash default)
[2025-06-05T17:14:24,293][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000` (logstash default)
[2025-06-05T17:14:24,293][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-nesting-depth` configured to `1000` (logstash default)
[2025-06-05T17:14:24,307][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because command line options are specified
[2025-06-05T17:14:24,560][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2025-06-05T17:14:24,817][INFO ][org.reflections.Reflections] Reflections took 48 ms to scan 1 urls, producing 149 keys and 521 values
[2025-06-05T17:14:24,863][INFO ][logstash.codecs.json     ] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2025-06-05T17:14:25,008][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2025-06-05T17:14:25,013][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1"]}
[2025-06-05T17:14:25,073][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://{cloud-host}.us-west-2.aws.found.io:443/]}}
[2025-06-05T17:14:25,278][INFO ][logstash.outputs.elasticsearch][main] Connected to ES instance {:url=>"https://{outpus-es-host}.us-west-2.aws.found.io:443/"}
[2025-06-05T17:14:25,279][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.17.0) {:es_version=>8}
[2025-06-05T17:14:25,430][INFO ][co.elastic.logstash.filters.elasticintegration.PreflightCheck][main] Connected to Elasticsearch version: 8.17.0
[2025-06-05T17:14:25,431][INFO ][co.elastic.logstash.filters.elasticintegration.PreflightCheck][main] Elasticsearch build_flavor: default
[2025-06-05T17:14:25,433][INFO ][logstash.filters.elasticintegration][main] by not manually configuring self-managed databases with `geoip_database_directory => ...` you accept and agree to the MaxMind EULA, which allows Elastic Integrations to use Logstash's Geoip Database Management service. For more details please visit https://www.maxmind.com/en/geolite2/eula
[2025-06-05T17:14:26,268][INFO ][co.elastic.logstash.filters.elasticintegration.PreflightCheck][main] Elasticsearch license OK (active enterprise)
[2025-06-05T17:14:26,268][INFO ][logstash.filters.elasticintegration][main] This 9.0.0 version of plugin embedded Ingest node components from Elasticsearch 9.0
[2025-06-05T17:14:26,268][WARN ][logstash.filters.elasticintegration][main] This plugin v9.0.0 is connected to an older MAJOR version of Elasticsearch v8.17.0, and may have trouble loading or running pipelines that use features that were deprecated before Elasticsearch v9.0; for the best experience, align major/minor versions across the Elastic Stack.
[2025-06-05T17:14:26,274][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1250, "pipeline.sources"=>["/Users/mashhur/Dev/elastic/logstash/config/elastic_integration_simple.conf"], :thread=>"#<Thread:0xf0a96db /Users/mashhur/Dev/elastic/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}
[2025-06-05T17:14:26,608][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.33}
[2025-06-05T17:14:26,625][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2025-06-05T17:14:26,636][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
       "sequence" => 0,
     "@timestamp" => 2025-06-06T00:14:26.634535Z,
    "data_stream" => {
        "namespace" => "default",
             "type" => "logs",
          "dataset" => "citrix_waf.log"
    },
      "@metadata" => {
        "target_ingest_pipeline" => "_none",
              "_ingest_document" => {
            "timestamp" => "2025-06-06T00:14:26.634535Z",
                "index" => "logs-citrix_waf.log-default"
        }
    },
           "host" => "Mashhurs-MacBook-Pro.local",
       "@version" => "1",
        "message" => "Dec 18 21:46:17 <local0.info>{my-message} spt=1 method=GET",
          "event" => {
        "original" => "{\n                \"data_stream\": {\n                    \"type\": \"logs\",\n                    \"dataset\": \"citrix_waf.log\",\n                    \"namespace\": \"default\"\n                },\n                \"message\": \"Dec 18 21:46:17 <local0.info> {redacted}\"\n             }"
    }
}
{
       "sequence" => 1,
     "@timestamp" => 2025-06-06T00:14:26.635851Z,
    "data_stream" => {
        "namespace" => "default",
             "type" => "logs",
          "dataset" => "citrix_waf.log"
    },
      "@metadata" => {
        "target_ingest_pipeline" => "_none",
              "_ingest_document" => {
            "timestamp" => "2025-06-06T00:14:26.635851Z",
                "index" => "logs-citrix_waf.log-default"
        }
    },
           "host" => "Mashhurs-MacBook-Pro.local",
       "@version" => "1",
        "message" => "{redacted}",
          "event" => {
        "original" => "{\n                \"data_stream\": {\n                    \"type\": \"logs\",\n                    \"dataset\": \"citrix_waf.log\",\n                    \"namespace\": \"default\"\n                },\n                \"message\": \"Dec 18 21:46:17 <local0.info> {redacted}\"\n             }"
    }
}
[2025-06-05T17:14:27,315][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2025-06-05T17:14:27,653][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2025-06-05T17:14:27,661][INFO ][logstash.runner          ] Logstash shut down.
╭─me ~/Dev/elastic/logstash  ‹upstream-9.0*› 
╰─➤

and we can check in apache-http debug logs:

[2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.client.InternalHttpAsyncClient][main] [exchange: 1] Connection allocated: CPoolProxy{http-outgoing-0 [ACTIVE]}
[2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl][main] http-outgoing-0 127.0.0.1:51598<->127.0.0.1:8083[ACTIVE][r:]: Set attribute http.nio.exchange-handler
[2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl][main] http-outgoing-0 127.0.0.1:51598<->127.0.0.1:8083[ACTIVE][rw:]: Event set [w]
[2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl][main] http-outgoing-0 127.0.0.1:51598<->127.0.0.1:8083[ACTIVE][rw:]: Set timeout 0
[2025-06-05T17:21:12,843][DEBUG][org.apache.http.impl.nio.client.InternalIODispatch][main] http-outgoing-0 [ACTIVE]: Connected
[2025-06-05T17:21:12,844][DEBUG][org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionImpl][main] http-outgoing-0 127.0.0.1:51598<->127.0.0.1:8083[ACTIVE][rw:]: Set attribute http.nio.http-exchange-state
****

mergify · 2025-06-06T00:23:27Z

This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

mashhurs · 2025-06-06T00:47:34Z

No anymore valid.

---- HISTORY ----
Failed CIs are not because of changes:

Issue-1: known Gem::MissingSpecError

  | 2025-06-05 17:25:59 PDT | #27 [logstash  7/13] RUN gem install bundler -v '< 2'
  | 2025-06-05 17:26:04 PDT | #27 5.005 ERROR:  Loading command: install (Gem::MissingSpecError)
  | 2025-06-05 17:26:04 PDT | #27 5.005 	Gem::MissingSpecError
  | 2025-06-05 17:26:04 PDT | #27 5.008 	/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/rubygems/specification.rb:1453:in `block in activate_dependencies'
  | 2025-06-05 17:26:04 PDT | #27 5.008 	org/jruby/RubyArray.java:1981:in `each'
...

Issue-2: artifact not found

  | 2025-06-05 17:25:24 PDT | Testing against version: 9.0.3-SNAPSHOT (distribution: default)
  | 2025-06-05 17:25:24 PDT | Checking manifest for docker.elastic.co/logstash/logstash:9.0.3-SNAPSHOT
  | 2025-06-05 17:25:24 PDT | no such manifest: docker.elastic.co/logstash/logstash:9.0.3-SNAPSHOT

robbavey · 2025-06-06T20:48:55Z

src/test/java/co/elastic/logstash/filters/elasticintegration/geoip/IpDatabaseProviderTest.java


    static void withGeoipProcessor(final IpDatabaseProvider geoIpDatabaseProvider, Map<String, Object> config, ExceptionalConsumer<Processor> geoIpProcessorConsumer) throws Exception {
-        Processor processor = new GeoIpProcessor.Factory("geoip", geoIpDatabaseProvider).create(Map.of(), null, null, config);
+        Processor processor = new GeoIpProcessor.Factory("geoip", geoIpDatabaseProvider).create(Map.of(), null, null, config, null);


Was this previously failing to compile?

Here is the answer: #305

Can you please split this into a separate commit?

Sorry missed addressing your request.
I have separated this change into this PR to see and discuss to apply Java unit tests run. Let's move there to continue this topic.

robbavey · 2025-06-24T12:40:20Z

buildkite test this

docs/index.asciidoc

robbavey · 2025-06-24T19:36:48Z

src/test/java/co/elastic/logstash/filters/elasticintegration/geoip/IpDatabaseProviderTest.java


    static void withGeoipProcessor(final IpDatabaseProvider geoIpDatabaseProvider, Map<String, Object> config, ExceptionalConsumer<Processor> geoIpProcessorConsumer) throws Exception {
-        Processor processor = new GeoIpProcessor.Factory("geoip", geoIpDatabaseProvider).create(Map.of(), null, null, config);
+        Processor processor = new GeoIpProcessor.Factory("geoip", geoIpDatabaseProvider).create(Map.of(), null, null, config, null);


Can you please split this into a separate commit?

robbavey · 2025-06-25T12:34:42Z

I will LGTM this as soon as #306 is merged and this PR is rebased against it

… the setting to ES client. URL is an uri LS type.

elasticmachine · 2025-06-26T22:24:42Z

💔 Build Failed

Buildkite Build
Commit: b974aef

Failed CI Steps

History

💔 Build #561 failed f9c6042
💔 Build #554 failed 0d9a968
💔 Build #552 failed 09a47da
💔 Build #551 failed 09a47da

robbavey

LGTM

mashhurs · 2025-06-27T14:31:21Z

@Mergifyio backport 9.1

github-actions · 2025-06-27T14:31:23Z

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

mergify · 2025-06-27T14:31:26Z

backport 9.1

✅ Backports have been created

#316 [9.1] (backport #304) Proxy support has been created for branch 9.1

mergify · 2025-06-27T14:31:46Z

backport 8.17 8.18 8.19 9.0 9.1

✅ Backports have been created

#317 [8.17] (backport #304) Proxy support has been created for branch 8.17
#318 [8.18] (backport #304) Proxy support has been created for branch 8.18
#319 [8.19] (backport #304) Proxy support has been created for branch 8.19
#320 [9.0] (backport #304) Proxy support has been created for branch 9.0
#316 [9.1] (backport #304) Proxy support has been created for branch 9.1

… the setting to ES client. URL is an uri LS type. (#304) (cherry picked from commit 2d89bc0)

… the setting to ES client. URL is an uri LS type. (#304) (#317) (cherry picked from commit 2d89bc0) Co-authored-by: Mashhur <[email protected]>

… the setting to ES client. URL is an uri LS type. (#304) (#318) (cherry picked from commit 2d89bc0) Co-authored-by: Mashhur <[email protected]>

… the setting to ES client. URL is an uri LS type. (#304) (#319) (cherry picked from commit 2d89bc0) Co-authored-by: Mashhur <[email protected]>

… the setting to ES client. URL is an uri LS type. (#304) (#320) (cherry picked from commit 2d89bc0) Co-authored-by: Mashhur <[email protected]>

… the setting to ES client. URL is an uri LS type. (#304) (#316) (cherry picked from commit 2d89bc0) Co-authored-by: Mashhur <[email protected]>

mashhurs added backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-active-8 Automated backport with mergify to all the active 8.[0-9]+ branches labels Jun 6, 2025

robbavey reviewed Jun 6, 2025

View reviewed changes

mashhurs mentioned this pull request Jun 24, 2025

[Feature]: Add proxy support for the Elastic Integration Filter #303

Closed

robbavey requested changes Jun 24, 2025

View reviewed changes

mashhurs requested a review from robbavey June 25, 2025 04:53

mashhurs force-pushed the proxy-support branch from 0d9a968 to f9c6042 Compare June 26, 2025 21:40

Proxy support: introduces proxy param to setup a proxy URL which sets…

b974aef

… the setting to ES client. URL is an uri LS type.

mashhurs force-pushed the proxy-support branch from f9c6042 to b974aef Compare June 26, 2025 22:22

robbavey approved these changes Jun 27, 2025

View reviewed changes

mashhurs merged commit 2d89bc0 into elastic:main Jun 27, 2025
3 of 4 checks passed

mashhurs deleted the proxy-support branch June 27, 2025 14:31

mergify bot pushed a commit that referenced this pull request Jun 27, 2025

Proxy support: introduces proxy param to setup a proxy URL which sets…

007080b

… the setting to ES client. URL is an uri LS type. (#304) (cherry picked from commit 2d89bc0)

mergify bot pushed a commit that referenced this pull request Jun 27, 2025

Proxy support: introduces proxy param to setup a proxy URL which sets…

012fa6c

… the setting to ES client. URL is an uri LS type. (#304) (cherry picked from commit 2d89bc0)

mergify bot pushed a commit that referenced this pull request Jun 27, 2025

Proxy support: introduces proxy param to setup a proxy URL which sets…

71719d0

… the setting to ES client. URL is an uri LS type. (#304) (cherry picked from commit 2d89bc0)

mergify bot pushed a commit that referenced this pull request Jun 27, 2025

Proxy support: introduces proxy param to setup a proxy URL which sets…

a74f9e8

… the setting to ES client. URL is an uri LS type. (#304) (cherry picked from commit 2d89bc0)

mergify bot pushed a commit that referenced this pull request Jun 27, 2025

Proxy support: introduces proxy param to setup a proxy URL which sets…

8ab17b1

… the setting to ES client. URL is an uri LS type. (#304) (cherry picked from commit 2d89bc0)

Proxy support #304

Proxy support #304

Uh oh!

Conversation

mashhurs commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test

Uh oh!

mergify bot commented Jun 6, 2025

Uh oh!

mashhurs commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robbavey Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

mashhurs Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

robbavey Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

mashhurs Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

robbavey commented Jun 24, 2025

Uh oh!

Uh oh!

robbavey Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

robbavey commented Jun 25, 2025

Uh oh!

elasticmachine commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

robbavey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mashhurs commented Jun 27, 2025

Uh oh!

github-actions bot commented Jun 27, 2025

Uh oh!

mergify bot commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify bot commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mashhurs commented Jun 6, 2025 •

edited

Loading

mashhurs commented Jun 6, 2025 •

edited

Loading

elasticmachine commented Jun 26, 2025 •

edited

Loading

mergify bot commented Jun 27, 2025 •

edited

Loading

mergify bot commented Jun 27, 2025 •

edited

Loading