Commit 078ebce
authored
[UNOMI-828] Support for OpenSearch persistence (#715)
* Minor Quickstart refactoring
* Merge changes from master branch
* Additional remove commands for actions, conditions, sessions, rules to be able to fix problems with deployments
* Initial working OpenSearch implementation. Basic functionality is working, work on automated tests is still in progress.
* Merges from master
* Bump all versions to 2.7
* Fix interface in blueprint descriptor
* Fix interface in blueprint descriptor
* Fix interface in blueprint descriptor
* - Fix issues with integration tests, ES tests now work 100%
- Changed Unomi startup to use features instead of bundles
- Added new build script that integrates all the functionality of the other build scripts
- Fix IPv6 address parsing
- Merge latest changes from master branch
-
* - Remove old build scripts
- Add option to new build script to be able to use opensearch with integration tests
* - Add new conditions bundle for ElasticSearch-specific conditions
- Replace hardcoded past event ES query builder in pasteventconditionbuilder to use a generic interface
- Replace GeoDistance and DistanceUnit used directly from ElasticSearch with clean-room implementation that are validated through unit tests. Also added unit tests on the ElasticSearch implementation to be able to test any differences
- Also added clean-room implementation of the DateMathParser so that it can be used with ElasticSearch or OpenSearch
- Modified Unomi startup mechanism to use features instead of bundles. Unfortunately due to complex interdependencies between bundles the features could not be split as wanted, so there is some duplication in the list of bundles.
- Removed last of inter-dependencies in the base plugin and the persistence-spi to try to resolve bundle inter-dependencies but that wasn't enough.
- ElasticSearch integration tests are now execute without any errors !
* Work on making integration tests work with OpenSearch:
- Removed elasticsearch-core from bundle watch requirements
- Fix issues with date parsing due to case sensitivity
- Improved test units for date parsing and date math handling
- Modified HealthChecks to provide an OpenSearch check provider (not yet fully working)
- Deactivate 1.x to 2.x migration integration test for OpenSearch (No OpenSearch users will be coming from 1.x)
- Update OpenSearch past event query builder to latest changes done in ElasticSearch past event query builder
- Various fixes in the integration tests to make them compatible with OpenSearch (removed hardcoded elasticsearch configuration and references)
- Added new shell script in itests directory to make it easier to handle the dynamically generated Pax Exam Karaf test container directory. Documentation is also included in the README file inside the itests directory.
* Work on making integration tests work with OpenSearch:
- Removed elasticsearch-core from bundle watch requirements
- Fix issues with date parsing due to case sensitivity
- Improved test units for date parsing and date math handling
- Modified HealthChecks to provide an OpenSearch check provider (not yet fully working)
- Deactivate 1.x to 2.x migration integration test for OpenSearch (No OpenSearch users will be coming from 1.x)
- Update OpenSearch past event query builder to latest changes done in ElasticSearch past event query builder
- Various fixes in the integration tests to make them compatible with OpenSearch (removed hardcoded elasticsearch configuration and references)
- Added new shell script in itests directory to make it easier to handle the dynamically generated Pax Exam Karaf test container directory. Documentation is also included in the README file inside the itests directory.
* Add missing ASL header
* - Introduce new ProgressListener system to indicate the current progress status in the integration tests
- Make sure the Unomi Management Service is started in IT tests before starting the unomi:start command
- Add support for minimal cluster state to allow to start an OpenSearch cluster with yellow status in IT tests
- Fix OpenSearch configuration prefix
- Modify HealthCheck providers to only be available depending on the availability of the persistence implementation.
- Fix integration tests to work properly with OpenSearch.
- Fix OpenSearch persistence initial startup
- Restructure startFeatures configuration to use arrays instead of complex parsing
- Modify OpenSearch custom object mapping to serialize map entries that have null values (which is the default for the ElasticSearch implementation).
-
* - Introduce new ProgressListener system to indicate the current progress status in the integration tests
- Make sure the Unomi Management Service is started in IT tests before starting the unomi:start command
- Add support for minimal cluster state to allow to start an OpenSearch cluster with yellow status in IT tests
- Fix OpenSearch configuration prefix
- Modify HealthCheck providers to only be available depending on the availability of the persistence implementation.
- Fix integration tests to work properly with OpenSearch.
- Fix OpenSearch persistence initial startup
- Restructure startFeatures configuration to use arrays instead of complex parsing
- Modify OpenSearch custom object mapping to serialize map entries that have null values (which is the default for the ElasticSearch implementation).
- Make sure the OpenSearch docker container used for the IT tests is replaced when tests are restarted.
- Fix the handling of the OffsetDateTime in the OpenSearch Property condition query builder
- Fix the rule service IT to generate rules with proper conditions and actions
* - Small cosmetic changes to the progress listener's top 10 slowest tests output to be CSV compatible
- Added a known issue in the itests README to reference the log issue on OpenSearch 2.18.
* - Add auto-start and no-karaf options to build script
* - Fix NO_COLOR handling
* - Add support for OpenSearch in docker images
- Add docker compose support for OpenSearch
- Fix startup issues with updates to UnomiManagementService
- Documentation updates to add OpenSearch information (still to be completed)
* - Improve plugin documentation to explain how to implement plugins for both OpenSearch and ElasticSearch
- Update Health check README to explain how it now works with both ElasticSearch and OpenSearch engines
* - Add support for OpenSearch in docker images
- Add docker compose support for OpenSearch
- Fix startup issues with updates to UnomiManagementService
- Documentation updates to add OpenSearch information (still to be completed)
* Change max parallel stragegy to 1 to avoid port conflicts
* - Make the search port configurable so that we can avoid conflicts between the ElasticSearch and OpenSearch integration tests
- Add documentation on how to migrate from ElasticSearch to OpenSearch (not tested yet)
* Update persistence-opensearch/conditions/src/main/java/org/apache/unomi/persistence/opensearch/conditions/PropertyConditionOSQueryBuilder.java
* Update persistence-opensearch/conditions/src/main/java/org/apache/unomi/persistence/opensearch/conditions/PropertyConditionOSQueryBuilder.java
* Update persistence-opensearch/conditions/src/main/java/org/apache/unomi/persistence/opensearch/conditions/PropertyConditionOSQueryBuilder.java
* Update persistence-opensearch/core/src/main/java/org/apache/unomi/persistence/opensearch/OpenSearchPersistenceServiceImpl.java
* - Remove Claude config file
- Update documentation to indicate support for OpenSearch 3 instead of 2
* Fix some minor dependencies
* Make sure all OpenSearch documentation points to v3
* Fix Unomi version in documentation
* Refactor entrypoint.sh to build node URLs array for both OpenSearch and Elasticsearch configurations
* Added a maximum number of retries to address issued raised during code review.
* To address issue in code review, made sure we output to both the logger and the System.out systematically
* Enhance ProgressListener with detailed JUnit test run reporting features. Added visual elements, timing information, motivational quotes, and CSV output for performance data. Improved ANSI color support and structured documentation for better usability.
* Enhance ProgressSuite with detailed documentation and improved test method counting. Added support for nested test classes, real-time progress reporting, and thread-safe tracking of completed tests. Updated class structure to facilitate better integration with ProgressListener.
* Update RuleServiceIT to clarify default condition and action settings in rule creation
* Update healthcheck configuration to correct HTTP client settings for OpenSearch and Elasticsearch, including trust certificate options and minimal cluster state defaults.
* Update Unomi version in quickstart and migration documentation from 2.0.0 to 3.0.0
* Clarify integration test impact of Maven profile selection for OpenSearch in configuration documentation
* Update setenv.sh to modify KARAF_OPTS for OpenSearch integration and remove auto-start option
* Refactor PropertyConditionEvaluator to use foldToASCII for string comparison, aligning with analyzer configuration behavior
* Revert IPv6 fix
* Refactor monthly index properties to rollover properties in configuration files and update related test cases. Rename test method for clarity and adjust logging messages to reflect changes in index template creation.
* Quick fix on description
* Refactor error handling in HealthCheckIT and OpenSearchPersistenceServiceImpl to use logging instead of printStackTrace.
Update exception messages in IdsConditionESQueryBuilder and IdsConditionOSQueryBuilder to include maximum IDs query count for better clarity.
Other general naming cleanup.
* Revert test unit runner to default one, will introduce new ProgressListener in a separate PR
* - Rename all query builder IDs to no longer use a reference to ElasticSearch
- Remove hover event query builder that is replaced with a condition definition with a parent condition
- Add a JSON schema for the hover event type
- Add missing JavaDocs
* Add detailed Javadoc comments to aggregation and condition interfaces for improved documentation
* - Removed non OpenSearch specific changes
- Added missing Javadoc comments
- Minor whitespace cleanups
* Update CI workflow to use actions/setup-java@v4 and improve OpenSearch integration test instructions in documentation
* Implement legacy query builder ID mapping for backward compatibility in Elasticsearch and OpenSearch. Add integration tests for legacy query builder functionality, including new condition definitions and JSON files for legacy conditions. Update documentation to reflect changes in query builder ID conventions and migration steps from previous versions.
* Refactor ConditionESQueryBuilderDispatcher and ConditionOSQueryBuilderDispatcher to utilize ConditionQueryBuilderDispatcherSupport for legacy ID resolution and contextualization. Remove hardcoded legacy ID mappings and improve documentation regarding legacy query builder handling.
* Run the build serially to avoid parallel interference
* Update new poms to version 3.1.0
* Refactor condition query builder dispatchers to extend a new base class. Update legacy query builder references to use the new centralized mapping and logging mechanism. This change enhances code maintainability and prepares for future improvements.1 parent b617d5c commit 078ebce
File tree
173 files changed
+9957
-835
lines changed- .github/workflows
- docker
- src/main/docker
- extensions
- groovy-actions/karaf-kar
- src/main/feature
- healthcheck
- src/main
- java/org/apache/unomi/healthcheck
- provider
- servlet
- resources
- router/router-karaf-feature
- src/main/feature
- salesforce-connector/karaf-kar
- src/main/feature
- weather-update/karaf-kar
- src/main/feature
- graphql
- cxs-impl/src/main/resources/META-INF/cxs/conditions
- karaf-feature
- src/main/feature
- itests
- src/test
- java/org/apache/unomi/itests
- migration
- resources
- conditions
- kar
- src/main/feature
- lifecycle-watcher/src/main
- java/org/apache/unomi/lifecycle
- resources/OSGI-INF/blueprint
- manual/src/main/asciidoc
- jsonSchema
- migrations
- package
- src/main/resources/etc
- persistence-elasticsearch
- conditions/src/main
- java/org/apache/unomi/persistence/elasticsearch/querybuilders/advanced
- resources/OSGI-INF/blueprint
- core
- src/main
- java/org/apache/unomi/persistence/elasticsearch
- resources/OSGI-INF/blueprint
- persistence-opensearch
- conditions
- src/main
- java/org/apache/unomi/persistence/opensearch/querybuilders/advanced
- resources/OSGI-INF/blueprint
- core
- src/main
- java/org/apache/unomi/persistence/opensearch
- querybuilders/core
- resources
- META-INF/cxs/mappings
- OSGI-INF/blueprint
- persistence-spi
- src
- main
- java/org/apache/unomi/persistence/spi
- aggregate
- conditions
- datemath
- dispatcher
- evaluator
- geo
- resources
- test/java/org/apache/unomi/persistence/spi/conditions
- datemath
- geo
- plugins
- baseplugin
- src/main
- java/org/apache/unomi/plugins/baseplugin/conditions
- resources
- META-INF/cxs/conditions
- OSGI-INF/blueprint
- hover-event
- src/main
- java/org/apache/unomi/plugins/events/hover/querybuilders
- resources
- META-INF/cxs
- conditions
- schemas
- rest
- services/src/main
- java/org/apache/unomi/services
- impl/cluster
- sorts
- resources/OSGI-INF/blueprint
- src/main/resources/assemblies
- tools/shell-commands
- src/main
- java/org/apache/unomi/shell
- actions
- migration
- service
- utils
- services
- internal
- resources
- wab/src/main/java/org/apache/unomi/web
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
173 files changed
+9957
-835
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
24 | 24 | | |
25 | | - | |
26 | | - | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| 33 | + | |
32 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
33 | 44 | | |
34 | 45 | | |
35 | 46 | | |
36 | | - | |
| 47 | + | |
37 | 48 | | |
38 | | - | |
39 | | - | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
40 | 52 | | |
41 | | - | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
42 | 62 | | |
43 | 63 | | |
44 | 64 | | |
45 | 65 | | |
46 | | - | |
| 66 | + | |
47 | 67 | | |
48 | 68 | | |
49 | 69 | | |
50 | 70 | | |
51 | 71 | | |
52 | | - | |
| 72 | + | |
53 | 73 | | |
54 | 74 | | |
55 | 75 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| |||
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
19 | | - | |
| 20 | + | |
| 21 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
| 42 | + | |
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
51 | 58 | | |
52 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
53 | 65 | | |
54 | | - | |
55 | | - | |
| 66 | + | |
56 | 67 | | |
57 | | - | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
58 | 78 | | |
59 | | - | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
60 | 90 | | |
61 | 91 | | |
62 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
63 | 108 | | |
64 | 109 | | |
65 | 110 | | |
66 | 111 | | |
67 | 112 | | |
68 | | - | |
| 113 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
98 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
99 | 101 | | |
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
103 | 105 | | |
104 | 106 | | |
105 | 107 | | |
106 | | - | |
107 | | - | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
108 | 112 | | |
109 | 113 | | |
110 | 114 | | |
| |||
133 | 137 | | |
134 | 138 | | |
135 | 139 | | |
136 | | - | |
| 140 | + | |
137 | 141 | | |
138 | 142 | | |
139 | 143 | | |
| |||
161 | 165 | | |
162 | 166 | | |
163 | 167 | | |
164 | | - | |
| 168 | + | |
165 | 169 | | |
166 | 170 | | |
167 | | - | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
168 | 179 | | |
169 | 180 | | |
170 | 181 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
24 | 31 | | |
| 32 | + | |
25 | 33 | | |
26 | 34 | | |
27 | 35 | | |
| |||
34 | 42 | | |
35 | 43 | | |
36 | 44 | | |
| 45 | + | |
37 | 46 | | |
38 | 47 | | |
39 | 48 | | |
| 49 | + | |
| 50 | + | |
40 | 51 | | |
41 | 52 | | |
Lines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
39 | 44 | | |
40 | 45 | | |
41 | 46 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
0 commit comments