Skip to content

Commit a4d6f46

Browse files
elahrvivazelahrvivaz
authored
GEOMESA-3545,GEOMESA-3035,GEOMESA-2816 Consolidate local and remote query planning (#3466)
* Consolidate local query runners to create query plans * For partially local query plans, move filtering and transforming from a reduce step to part of the query step * Also fixes: * GEOMESA-3035 Redis - Fix transforms in explain logging * GEOMESA-2816 HBase - Fix analytic scans with column groups when remote filtering is disabled
1 parent 50c26c4 commit a4d6f46

File tree

84 files changed

+1802
-1735
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

84 files changed

+1802
-1735
lines changed

‎.github/workflows/build-and-test.yml‎

Lines changed: 0 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -111,23 +111,3 @@ jobs:
111111
- name: Remove geomesa artifacts
112112
if: success() || failure()
113113
run: rm -rf ~/.m2/repository/org/locationtech/geomesa
114-
validate-ci:
115-
# validates that we haven't missed any projects in our build matrix
116-
name: validate-ci
117-
runs-on: ubuntu-latest
118-
if: github.event_name == 'pull_request'
119-
steps:
120-
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
121-
- name: Validate project list
122-
run: |
123-
while IFS= read -r -d '' pom; do
124-
grep -q 'packaging>pom' "$pom" || {
125-
module="${pom%/pom.xml}"
126-
module="${module#./}"
127-
if [[ $module != *distributed-runtime* ]]; then
128-
echo "Checking $module"
129-
yq -e '.jobs.build.strategy.matrix.projects[].list | select(test("(^|[, ])'"${module}"'($|[, ])"))' .github/workflows/build-and-test.yml >/dev/null 2>&1 || exit 1
130-
fi
131-
}
132-
done < <(find . -name pom.xml -print0)
133-
echo "done"

‎.github/workflows/integration-tests.yml‎

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -51,21 +51,3 @@ jobs:
5151
- name: Remove geomesa artifacts
5252
if: success() || failure()
5353
run: rm -rf ~/.m2/repository/org/locationtech/geomesa
54-
validate-ci:
55-
# validates that we haven't missed any projects in our build matrix
56-
name: validate-ci
57-
runs-on: ubuntu-latest
58-
if: github.event_name == 'pull_request'
59-
steps:
60-
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
61-
- name: Validate project list
62-
run: |
63-
while IFS= read -r -d '' test_class; do
64-
module="${test_class%/src/test/*}"
65-
module="${module#./}"
66-
if [[ $module != *spark ]]; then
67-
echo "Checking $module"
68-
yq -e '.jobs.integration-tests.strategy.matrix.projects[].list | select(test("(^|[, ])'"${module}"'($|[, ])"))' .github/workflows/integration-tests.yml >/dev/null 2>&1 || exit 1
69-
fi
70-
done < <(find . -name '*IT.scala' -print0)
71-
echo "done"

‎.github/workflows/javadocs.yml‎

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -79,21 +79,3 @@ jobs:
7979
- name: Remove geomesa artifacts
8080
if: success() || failure()
8181
run: rm -rf ~/.m2/repository/org/locationtech/geomesa
82-
validate-ci:
83-
# validates that we haven't missed any projects in our build matrix
84-
name: validate-ci
85-
runs-on: ubuntu-latest
86-
if: github.event_name == 'pull_request'
87-
steps:
88-
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
89-
- name: Validate project list
90-
run: |
91-
while IFS= read -r -d '' pom; do
92-
grep -q 'packaging>pom' "$pom" || {
93-
module="${pom%/pom.xml}"
94-
module="${module#./}"
95-
echo "Checking $module"
96-
yq -e '.jobs.javadocs.strategy.matrix.projects[].list | select(test("(^|[, ])'"${module}"'($|[, ])"))' .github/workflows/javadocs.yml >/dev/null 2>&1 || exit 1
97-
}
98-
done < <(find . -name pom.xml -print0)
99-
echo "done"

‎.github/workflows/spark.yml‎

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -55,21 +55,3 @@ jobs:
5555
- name: Remove geomesa artifacts
5656
if: success() || failure()
5757
run: rm -rf ~/.m2/repository/org/locationtech/geomesa
58-
validate-ci:
59-
# validates that we haven't missed any projects in our build matrix
60-
name: validate-ci
61-
runs-on: ubuntu-latest
62-
if: github.event_name == 'pull_request'
63-
steps:
64-
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
65-
- name: Validate project list
66-
run: |
67-
while IFS= read -r -d '' test_class; do
68-
module="${test_class%/src/test/*}"
69-
module="${module#./}"
70-
if [[ $module =~ .*spark ]]; then
71-
echo "Checking $module"
72-
yq -e '.jobs.integration-tests.strategy.matrix.projects[].list | select(test("(^|[, ])'"${module}"'($|[, ])"))' .github/workflows/spark.yml >/dev/null 2>&1 || exit 1
73-
fi
74-
done < <(find . -name '*IT.scala' -print0)
75-
echo "done"

‎.github/workflows/validate-ci.yml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# validates that we haven't missed any projects in our build matrices
2+
name: validate-ci
3+
4+
on:
5+
pull_request:
6+
7+
permissions: # added using https://github.com/step-security/secure-repo
8+
contents: read
9+
10+
env:
11+
JAVA_VERSION: 17
12+
MAVEN_CLI_OPTS: -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false --batch-mode -Dlicense.skip=true
13+
MAVEN_COMPILE_ARGS: clean install -Dmaven.test.skip -Dmaven.assembly.skip=true -Dmaven.source.skip -Pskip-spark-runtimes -T4
14+
MAVEN_TEST_ARGS: test -Dmaven.main.skip
15+
16+
jobs:
17+
validate-tests:
18+
runs-on: ubuntu-latest
19+
steps:
20+
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
21+
- name: Validate project list
22+
run: |
23+
while IFS= read -r -d '' pom; do
24+
grep -q 'packaging>pom' "$pom" || {
25+
module="${pom%/pom.xml}"
26+
module="${module#./}"
27+
if [[ $module != *distributed-runtime* ]]; then
28+
echo "Checking $module"
29+
yq -e '.jobs.build.strategy.matrix.projects[].list | select(test("(^|[, ])'"${module}"'($|[, ])"))' .github/workflows/build-and-test.yml >/dev/null 2>&1 || exit 1
30+
fi
31+
}
32+
done < <(find . -name pom.xml -print0)
33+
echo "done"
34+
validate-integration-tests:
35+
runs-on: ubuntu-latest
36+
steps:
37+
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
38+
- name: Validate project list
39+
run: |
40+
while IFS= read -r -d '' test_class; do
41+
module="${test_class%/src/test/*}"
42+
module="${module#./}"
43+
if [[ $module != *spark ]]; then
44+
echo "Checking $module"
45+
yq -e '.jobs.integration-tests.strategy.matrix.projects[].list | select(test("(^|[, ])'"${module}"'($|[, ])"))' .github/workflows/integration-tests.yml >/dev/null 2>&1 || exit 1
46+
fi
47+
done < <(find . -name '*IT.scala' -print0)
48+
echo "done"
49+
validate-javadocs:
50+
runs-on: ubuntu-latest
51+
steps:
52+
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
53+
- name: Validate project list
54+
run: |
55+
while IFS= read -r -d '' pom; do
56+
grep -q 'packaging>pom' "$pom" || {
57+
module="${pom%/pom.xml}"
58+
module="${module#./}"
59+
echo "Checking $module"
60+
yq -e '.jobs.javadocs.strategy.matrix.projects[].list | select(test("(^|[, ])'"${module}"'($|[, ])"))' .github/workflows/javadocs.yml >/dev/null 2>&1 || exit 1
61+
}
62+
done < <(find . -name pom.xml -print0)
63+
echo "done"
64+
validate-spark:
65+
runs-on: ubuntu-latest
66+
steps:
67+
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
68+
- name: Validate project list
69+
run: |
70+
while IFS= read -r -d '' test_class; do
71+
module="${test_class%/src/test/*}"
72+
module="${module#./}"
73+
if [[ $module =~ .*spark ]]; then
74+
echo "Checking $module"
75+
yq -e '.jobs.integration-tests.strategy.matrix.projects[].list | select(test("(^|[, ])'"${module}"'($|[, ])"))' .github/workflows/spark.yml >/dev/null 2>&1 || exit 1
76+
fi
77+
done < <(find . -name '*IT.scala' -print0)
78+
echo "done"

‎docs/tutorials/geomesa-quickstart-lambda.rst‎

Lines changed: 2 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -219,17 +219,9 @@ the feature moving around.
219219
Transient vs Persistent Features
220220
--------------------------------
221221

222-
The layer preview will merge the results of features from Kafka with features from Accumulo. You may disable
223-
results from one of the source by using the ``viewparams`` parameter:
224-
225-
.. code-block:: bash
226-
227-
...&viewparams=LAMBDA_QUERY_TRANSIENT:false
228-
...&viewparams=LAMBDA_QUERY_PERSISTENT:false
229-
222+
The layer preview will merge the results of features from Kafka with features from Accumulo.
230223
While the quick start is running, all the features should be returned from the transient store (Kafka). After the quick
231-
start finishes, all the feature should be returned from the persistent store (Accumulo). You can play with the
232-
``viewparams`` to see the difference.
224+
start finishes, all the feature should be returned from the persistent store (Accumulo).
233225

234226
Looking at the Code
235227
-------------------

‎docs/user/upgrade.rst‎

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -158,17 +158,6 @@ Internal API Changes
158158
GeoMesa does not have a well-defined public API. This section details classes and methods that have been changed or removed,
159159
but are not meant to be publicly available.
160160

161-
.. raw:: html
162-
163-
<details>
164-
<summary><b>Click here to see all API changes</b></summary>
165-
166-
Internal API Changes
167-
--------------------
168-
169-
GeoMesa does not have a well-defined public API. This section details classes and methods that have been changed or removed,
170-
but are not meant to be publicly available.
171-
172161
.. raw:: html
173162

174163
<details>
@@ -190,6 +179,7 @@ Relocated Classes
190179
* ``org.locationtech.geomesa.fs.storage.parquet.ParquetPathReader`` ->
191180
``org.locationtech.geomesa.fs.storage.parquet.io.ParquetFileSystemReader``
192181
* ``org.locationtech.geomesa.fs.storage.parquet.SimpleFeatureParquetWriter`` -> ``org.locationtech.geomesa.fs.storage.parquet.io``
182+
* ``org.locationtech.geomesa.index.planning.QueryPlanner.CostEvaluation`` -> ``org.locationtech.geomesa.index.planning.StrategyDecider.CostEvaluation``
193183
* ``org.locationtech.geomesa.utils.index.SizeSeparatedBucketIndex`` -> ``org.locationtech.geomesa.memory.index.impl``
194184
* ``org.locationtech.geomesa.utils.index.BucketIndex`` -> ``org.locationtech.geomesa.memory.index.impl``
195185
* ``org.locationtech.geomesa.utils.index.WrappedQuadTree`` -> ``org.locationtech.geomesa.memory.index.impl``
@@ -211,6 +201,7 @@ Removed Classes
211201
* ``org.locationtech.geomesa.filter.index.BucketIndexSupport``
212202
* ``org.locationtech.geomesa.filter.index.SizeSeparatedBucketIndexSupport``
213203
* ``org.locationtech.geomesa.fs.storage.common.AbstractFileSystemStorage.FileSystemPathReader``
204+
* ``org.locationtech.geomesa.index.planning.LocalQueryRunner.LocalTransformReducer``
214205
* ``org.locationtech.geomesa.metrics.micrometer.MicrometerSetup``
215206
* ``org.locationtech.geomesa.security.VisibilityEvaluator``
216207
* ``org.locationtech.geomesa.tools.utils.StorageJobUtils``
@@ -255,8 +246,10 @@ Removed Methods
255246
* ``org.locationtech.geomesa.fs.data.FileSystemDataStore#this(FileContext)``
256247
* ``org.locationtech.geomesa.fs.storage.common.partitions.DateTimeScheme#apply(DateTimeFormatter)``
257248
* ``org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory`` - various system properties
249+
* ``org.locationtech.geomesa.index.geotools.GeoMesaDataStore#queryPlanner``
258250
* ``org.locationtech.geomesa.index.planning.LocalQueryRunner#visible``
259-
* ``org.locationtech.geomesa.index.planning.QueryPlanner`` - "perThreadQueryHints"-related methods
251+
* ``org.locationtech.geomesa.index.planning.QueryPlanner`` - most static methods
252+
* ``org.locationtech.geomesa.index.planning.QueryRunner#configureDefaultQuery``
260253
* ``org.locationtech.geomesa.kafka.data.KafkaDataStore#producer(KafkaDataStoreConfig)``
261254
* ``org.locationtech.geomesa.kafka.utils.GeoMessageSerializer#apply(SimpleFeatureType, SerializationType, Boolean)``
262255
* ``org.locationtech.geomesa.lambda.stream.kafka.KafkaStore#topic``

‎geomesa-accumulo/geomesa-accumulo-datastore/src/main/scala/org/locationtech/geomesa/accumulo/audit/AccumuloAuditWriter.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ class AccumuloAuditWriter(
6767
user: String,
6868
filter: Filter,
6969
hints: Hints,
70-
plans: Seq[QueryPlan[_]],
70+
plans: Seq[QueryPlan],
7171
startTime: Long,
7272
endTime: Long,
7373
planTime: Long,

0 commit comments

Comments
 (0)