Skip to content

Provide a pure Java Elasticsearch spatial plugin #2388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jpolchlo wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Provide a pure Java Elasticsearch spatial plugin #2388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
df1ce2f
Add Java implementation of spatial shape deserialization for ES8 comp…
Feb 6, 2026
6ac229f
Implement ring-linestring intersections in Java
Feb 11, 2026
e02d8dd
Port polygon BR operations to pure Java
Feb 11, 2026
952ae58
Add Circle shape with intersection support
Feb 11, 2026
c70bb0f
Fill out complete shape intersection matrix
Feb 11, 2026
a4bddcf
Remove Clojure delegation from ShapeIntersections
Feb 11, 2026
3ec2184
Reorganize Java packages for better structure
Feb 11, 2026
5c043df
Fix reflection warnings in AOT compilation
Feb 11, 2026
27c9cbd
Fix review comments: correctness and validation issues
Feb 11, 2026
6cefc39
Add defensive programming: immutable collections and Ring role tracking
Feb 11, 2026
ce048ea
Cache Java rings in Clojure records for performance
Feb 11, 2026
2ae1cbd
Apply code review fixes: harden geometry edge cases
Feb 12, 2026
7b7fe81
Fix nil handling in OrdsInfoShapes for granules without spatial data
Feb 13, 2026
94df37b
Fix MBR coordinate system inconsistency in arc-line segment intersect…
Feb 13, 2026
ff70261
Implement the es-spatial-plugin in pure Java
Feb 12, 2026
834afa8
Fix Java compilation and update to Java 11 target
Feb 16, 2026
3ae7730
Remove defunct Clojure plugin code
Feb 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions es-spatial-plugin/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
ARG BASE_IMAGE_REPO=

FROM ${BASE_IMAGE_REPO}docker.elastic.co/elasticsearch/elasticsearch:8.18.7

USER root
RUN apt update && apt upgrade -y && apt clean

COPY target/cmr-es-spatial-plugin-0.1.0-SNAPSHOT.zip /tmp
RUN mkdir -p /var/lib/elasticsearch/tmp &&\
chown elasticsearch:elasticsearch /var/lib/elasticsearch/tmp &&\
chown elasticsearch:elasticsearch /usr/share/elasticsearch &&\
bin/elasticsearch-plugin install discovery-ec2 --batch &&\
bin/elasticsearch-plugin install file:///tmp/cmr-es-spatial-plugin-0.1.0-SNAPSHOT.zip --batch &&\
chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/plugins &&\
echo "root soft memlock unlimited" >> /etc/security/limits.conf &&\
echo "root hard memlock unlimited" >> /etc/security/limits.conf
Comment on lines +15 to +16
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Memlock limits are configured for the wrong user.

Limits are appended for root, but Elasticsearch runs as UID 1000 here, so the intended memlock setting will not apply at runtime.

🔧 Proposed fix 
-    echo "root soft memlock unlimited" >> /etc/security/limits.conf &&\
-    echo "root hard memlock unlimited" >> /etc/security/limits.conf
+    echo "elasticsearch soft memlock unlimited" >> /etc/security/limits.conf &&\
+    echo "elasticsearch hard memlock unlimited" >> /etc/security/limits.conf

Also applies to: 20-20

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@es-spatial-plugin/Dockerfile` around lines 15 - 16, The Dockerfile is
appending memlock limits for "root" but Elasticsearch runs as UID 1000, so the
settings never apply; change the two echo lines that write "root soft memlock
unlimited" and "root hard memlock unlimited" to target the Elasticsearch runtime
user (UID 1000) or use a wildcard entry (e.g., "*" or the actual username used
at runtime) so the memlock limits apply to the process that runs Elasticsearch;
update the two instances in the Dockerfile where those echoes appear
accordingly.


WORKDIR /

USER 1000
ENV HOME=/usr/share/elasticsearch
99 changes: 45 additions & 54 deletions es-spatial-plugin/project.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
last
(clojure.string/replace "\"" ""))))

(def plugin-jar-name
(str "target/cmr-es-spatial-plugin-" version ".jar"))

(def uberjar-name
(str "target/cmr-es-spatial-plugin-" version "-standalone.jar"))

Expand All @@ -23,76 +26,65 @@
(def es-deps-target-path
"es-deps")

(def elastic-version "7.17.25")
(defn get-list-of-dep-jars []
(let [all-jars (into [] (map #(.getName %) (.listFiles (clojure.java.io/file "target/lib"))))
;; Minimal set: only what spatial-lib Java code actually needs
allowed-prefixes ["clojure" ; Runtime (spatial-lib has compiled Clojure)
"cmr-spatial-lib" ; Main library
"jafama" ; Math library used by spatial calculations
"primitive-math" ; Math optimizations
"vectorz"]] ; Vector math library
(map #(str "target/lib/" %) (filter (fn [jar-name] (some (fn [prefix] (str/starts-with? jar-name prefix)) allowed-prefixes)) all-jars))))
Comment on lines +29 to +37
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# First, locate and read the full project.clj file
find . -name "project.clj" -path "*/es-spatial-plugin/*" | head -1 | xargs cat -n

Repository: nasa/Common-Metadata-Repository

Length of output: 9514

🏁 Script executed:

# Also check for any require or ns declarations at the top of the file
head -50 es-spatial-plugin/project.clj

Repository: nasa/Common-Metadata-Repository

Length of output: 2425

🏁 Script executed:

# Search for any aliasing of clojure.string or str namespace in the file
rg -n "(require|ns|:as str)" es-spatial-plugin/project.clj

Repository: nasa/Common-Metadata-Repository

Length of output: 1134

str/starts-with? is unresolved — replace with fully-qualified namespace.

Line 37 uses str/starts-with?, but str is not aliased. All other string operations in this file use clojure.string/... with full qualification (lines 4, 7-9). The :clj-kondo/ignore [:unresolved-namespace] directive on line 1 explicitly suppresses this warning, confirming the issue is known.

Replace line 37:

Fix 
-    (map #(str "target/lib/" %) (filter (fn [jar-name] (some (fn [prefix] (str/starts-with? jar-name prefix)) allowed-prefixes)) all-jars))))
+    (map #(str "target/lib/" %) (filter (fn [jar-name] (some (fn [prefix] (clojure.string/starts-with? jar-name prefix)) allowed-prefixes)) all-jars))))
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@es-spatial-plugin/project.clj` around lines 29 - 37, The unresolved use of
str/starts-with? inside get-list-of-dep-jars should be replaced with the
fully-qualified clojure.string/starts-with?; update the filter callback that
checks jar-name against allowed-prefixes to call clojure.string/starts-with? (or
require alias) so the referenced symbol is resolved and consistent with other
string usage around allowed-prefixes and get-list-of-dep-jars.


(defproject nasa-cmr/cmr-es-spatial-plugin "0.1.0-SNAPSHOT"
:description "A Elastic Search plugin that enables spatial search entirely within elastic."
:url "https://github.com/nasa/Common-Metadata-Repository/tree/master/es-spatial-plugin"
:java-source-paths ["src/java"]
:javac-options ["-target" "11" "-source" "11"]
:jvm-opts ^:replace ["-server"
"-Dclojure.compiler.direct-linking=true"]
:plugins [[lein-shell "0.5.0"]]
:profiles {:security {:plugins [[com.livingsocial/lein-dependency-check "1.4.1"]]
:dependency-check {:output-format [:all]
:suppression-file "resources/security/suppression.xml"}}
:provided {:dependencies [[nasa-cmr/cmr-common-lib "0.1.1-SNAPSHOT"
:exclusions [[com.fasterxml.jackson.core/jackson-core]
[com.fasterxml.jackson.dataformat/jackson-dataformat-cbor]
[com.fasterxml.jackson.dataformat/jackson-dataformat-smile]
[com.fasterxml.jackson.dataformat/jackson-dataformat-yaml]]]
[nasa-cmr/cmr-spatial-lib "0.1.0-SNAPSHOT"
:exclusions [[com.fasterxml.jackson.core/jackson-core]
[com.fasterxml.jackson.dataformat/jackson-dataformat-cbor]
[com.fasterxml.jackson.dataformat/jackson-dataformat-smile]
[com.fasterxml.jackson.dataformat/jackson-dataformat-yaml]]]
[org.elasticsearch/elasticsearch ~elastic-version]
[org.clojure/tools.reader "1.3.2"]
[org.yaml/snakeyaml "1.31"]]}
:provided {:dependencies [[nasa-cmr/cmr-common-lib "0.1.1-SNAPSHOT"]
[nasa-cmr/cmr-spatial-lib "0.1.0-SNAPSHOT"]
[org.elasticsearch/elasticsearch "8.18.7"]]}
:es-deps {:dependencies [[nasa-cmr/cmr-spatial-lib "0.1.0-SNAPSHOT"
;; These exclusions will be provided by elasticsearch.
:exclusions [[com.dadrox/quiet-slf4j]
[com.fasterxml.jackson.core/jackson-core]
[com.fasterxml.jackson.dataformat/jackson-dataformat-cbor]
[com.fasterxml.jackson.dataformat/jackson-dataformat-smile]
[com.fasterxml.jackson.dataformat/jackson-dataformat-yaml]
[commons-io]
[commons-codec]
[commons-logging]
[joda-time]
[org.ow2.asm/asm]
[org.ow2.asm/asm-all]
;; Both lz4 libraries are linked together. yawk
;; is supposed to be a drop in replacement and
;; uses the same package name as the original.
[net.jpountz.lz4/lz4]
[at.yawk.lz4/lz4-java]
[org.locationtech.jts/jts-core]
[org.locationtech.jts.JTSVersion]
[org.slf4j/slf4j-api]]]
[org.clojure/tools.reader "1.3.2"]
[org.clojure/tools.reader "1.5.0"]
[org.clojure/clojure "1.11.2"]]
:target-path ~es-deps-target-path
:uberjar-name ~es-deps-uberjar-name
:jar-name ~es-deps-jar-name
:aot []}
:es-plugin {:aot [cmr.elasticsearch.plugins.spatial.script.core
cmr.elasticsearch.plugins.spatial.factory.lfactory
cmr.elasticsearch.plugins.spatial.factory.core
cmr.elasticsearch.plugins.spatial.engine.core
cmr.elasticsearch.plugins.spatial.plugin]}
:jar-deps {:plugins [[org.clojars.jj/copy-deps "1.0.1"]]
:dependencies [[nasa-cmr/cmr-spatial-lib "0.1.0-SNAPSHOT"]
[nasa-cmr/cmr-common-lib "0.1.1-SNAPSHOT"]]
:aot []}
:dev {:dependencies [[criterium "0.4.4"]
[cheshire "5.12.0"]
[org.clojure/tools.reader "1.3.2"]
[cheshire "5.13.0"]
[nasa-cmr/cmr-common-lib "0.1.1-SNAPSHOT"]
[nasa-cmr/cmr-spatial-lib "0.1.0-SNAPSHOT"]
[org.elasticsearch/elasticsearch ~elastic-version]
[org.elasticsearch/elasticsearch "8.18.7"]
[org.clojars.gjahad/debug-repl "0.3.3"]
[org.clojure/tools.nrepl "0.2.13"]
[org.clojure/tools.namespace "0.2.11"]
[org.yaml/snakeyaml "1.31"]]
:aot [cmr.elasticsearch.plugins.spatial.script.core
cmr.elasticsearch.plugins.spatial.factory.lfactory
cmr.elasticsearch.plugins.spatial.factory.core
cmr.elasticsearch.plugins.spatial.engine.core
cmr.elasticsearch.plugins.spatial.plugin]
[nrepl/nrepl "1.3.0"]
[org.clojure/tools.namespace "1.2.0"]]
:global-vars {*warn-on-reflection* false
*assert* false}}
:static {}
Expand All @@ -108,28 +100,27 @@
:kaocha {:dependencies [[lambdaisland/kaocha "1.0.732"]
[lambdaisland/kaocha-cloverage "1.0.75"]
[lambdaisland/kaocha-junit-xml "0.0.76"]]}}
:aliases {"install-es-deps" ["do"
"with-profile" "es-deps,provided" "clean,"
"with-profile" "es-deps,provided" "uberjar,"
;; target-path is being ignored for uberjar. move uberjar to es-deps-target-path.
["shell" "echo" "inst-es-deps"]
"shell" "mv" ~(str "target/" es-deps-uberjar-name) ~es-deps-target-path]
"install-es-plugin" ["do"
["shell" "echo" "inst-es-plugin"]
"with-profile" "es-plugin,provided" "clean,"
"with-profile" "es-plugin,provided" "uberjar,"]
"package-es-plugin" ["do"
:aliases {"install-es-plugin" ["do"
["shell" "echo" "Building ES spatial plugin JAR"]
"with-profile" "provided" "clean,"
"with-profile" "provided" "jar,"]
"gather-dependencies" ["do"
["shell" "echo" "Collecting dependent JARs"]
"with-profile" "jar-deps" "copy-deps,"]
"prepare-es-plugin" ["do"
"install-es-plugin"
["shell" "echo" "pack-es-deps"]
"shell"
"zip"
"-j"
~plugin-zip-name
~uberjar-name
"resources/plugin/plugin-descriptor.properties"]
"gather-dependencies"]
"package-es-plugin" ~(vec (concat ["do"
["shell" "echo" "Packaging ES plugin into zip file"]
"shell"
"zip"
"-j"
plugin-zip-name
plugin-jar-name
"resources/plugin/plugin-descriptor.properties"]
(get-list-of-dep-jars)))
Comment on lines +113 to +121
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

get-list-of-dep-jars is evaluated at project load time — jar list may be empty.

The ~(vec (concat ...)) unquote on line 113 evaluates (get-list-of-dep-jars) when project.clj is read, not when the package-es-plugin alias is executed. If target/lib doesn't exist yet (e.g., before gather-dependencies has ever been run), the jar list will be empty, and the zip will be missing dependency jars.

This means prepare-es-plugin and package-es-plugin must be run as separate Leiningen invocations (so project.clj is re-read after target/lib is populated). Running them in a single lein do prepare-es-plugin, package-es-plugin won't work correctly.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@es-spatial-plugin/project.clj` around lines 113 - 121, The alias
package-es-plugin currently unquotes ~(vec (concat ... (get-list-of-dep-jars)))
which evaluates get-list-of-dep-jars at project read time and can yield an empty
list; change package-es-plugin so the dependency jars are resolved at runtime
instead of when project.clj is loaded — for example replace the evaluated vector
with a runtime shell command (keep the "do" steps but use a single "shell" that
runs bash -lc and globs target/lib/*.jar, e.g. "bash -lc 'zip -j
<plugin-zip-name> <plugin-jar-name>
resources/plugin/plugin-descriptor.properties target/lib/*.jar'"), or implement
package-es-plugin as a small Leiningen task function that calls
get-list-of-dep-jars at invocation time; adjust references to package-es-plugin
and get-list-of-dep-jars accordingly.

"build-all" ["do"
["shell" "echo" "build-all"]
"install-es-deps,"
"install-es-plugin,"]

;; Kaocha test aliases
Expand Down
2 changes: 1 addition & 1 deletion es-spatial-plugin/resources/plugin/plugin-descriptor.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ classname=cmr.elasticsearch.plugins.SpatialSearchPlugin
# use the system property java.specification.version
# version string must be a sequence of nonnegative decimal integers
# separated by "."'s and may have leading zeros
java.version=1.8
java.version=11
#
# 'elasticsearch.version': version of elasticsearch compiled against
elasticsearch.version=7.17.14
Comment on lines +16 to 19
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# First, let's check if the files mentioned in the review exist and examine their contents
find . -type f -path "*es-spatial-plugin/resources/plugin/plugin-descriptor.properties" 2>/dev/null | head -5

Repository: nasa/Common-Metadata-Repository

Length of output: 140

🏁 Script executed:

# Search for version declarations in key plugin files
rg -n "java\.version|elasticsearch\.version" --type properties 2>/dev/null | head -20

Repository: nasa/Common-Metadata-Repository

Length of output: 57

🏁 Script executed:

# Check the Dockerfile for Elasticsearch version
find . -type f -name "Dockerfile" -path "*es-spatial-plugin*" 2>/dev/null | xargs cat 2>/dev/null

Repository: nasa/Common-Metadata-Repository

Length of output: 907

🏁 Script executed:

# Check project.clj or build files for version declarations
find . -type f -path "*es-spatial-plugin*" \( -name "project.clj" -o -name "*.gradle" -o -name "pom.xml" \) 2>/dev/null | xargs ls -lh 2>/dev/null

Repository: nasa/Common-Metadata-Repository

Length of output: 155

🏁 Script executed:

cat -n ./es-spatial-plugin/resources/plugin/plugin-descriptor.properties

Repository: nasa/Common-Metadata-Repository

Length of output: 794

🏁 Script executed:

cat -n ./es-spatial-plugin/project.clj

Repository: nasa/Common-Metadata-Repository

Length of output: 9514

Update plugin descriptor Elasticsearch version to match build target.

Line 19 declares elasticsearch.version=7.17.14 but this PR builds against Elasticsearch 8.18.7 (as specified in project.clj line 52 and the Dockerfile). Elasticsearch enforces version compatibility at plugin install time, so the mismatch will cause the plugin to fail loading.

Update the descriptor to match:

Fix 
-elasticsearch.version=7.17.14
+elasticsearch.version=8.18.7
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
java.version=11
#
# 'elasticsearch.version': version of elasticsearch compiled against
elasticsearch.version=7.17.14
java.version=11
#
# 'elasticsearch.version': version of elasticsearch compiled against
elasticsearch.version=8.18.7
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@es-spatial-plugin/resources/plugin/plugin-descriptor.properties` around lines
16 - 19, Update the plugin descriptor property elasticsearch.version in
plugin-descriptor.properties to match the build target (Elasticsearch 8.18.7);
locate the existing line "elasticsearch.version=7.17.14" and change its value to
"8.18.7" so the plugin descriptor aligns with project.clj/Dockerfile and avoids
install-time compatibility failures.

6 changes: 0 additions & 6 deletions es-spatial-plugin/resources/plugin/plugin-security.policy
View file
Open in desktop

This file was deleted.

41 changes: 0 additions & 41 deletions es-spatial-plugin/src/cmr/elasticsearch/plugins/spatial/engine/core.clj
View file
Open in desktop

This file was deleted.

22 changes: 0 additions & 22 deletions es-spatial-plugin/src/cmr/elasticsearch/plugins/spatial/factory/core.clj
View file
Open in desktop

This file was deleted.

90 changes: 0 additions & 90 deletions es-spatial-plugin/src/cmr/elasticsearch/plugins/spatial/factory/lfactory.clj
View file
Open in desktop

This file was deleted.

22 changes: 0 additions & 22 deletions es-spatial-plugin/src/cmr/elasticsearch/plugins/spatial/plugin.clj
View file
Open in desktop

This file was deleted.

Loading