8.19 add hugging face rerank support backport #128358
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o 1c7f5aa (elastic#126854) This PR contains the following updates: | Package | Update | Change | |---|---|---| | docker.elastic.co/wolfi/chainguard-base | digest | `29150cd` -> `1c7f5aa` | --- ### Configuration 📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://redirect.github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoiOC54IiwibGFiZWxzIjpbIjpEZWxpdmVyeS9QYWNrYWdpbmciLCI+bm9uLWlzc3VlIiwiVGVhbTpEZWxpdmVyeSIsImF1dG8tbWVyZ2Utd2l0aG91dC1hcHByb3ZhbCJdfQ==-->
Some basic coverage for ES|QL security with failure store access. The behavior w.r.t. status codes diverges quite a bit between ES|QL and _search but not in any failure store specific way. ES|QL has the same authorization behavior when querying multiple data streams without failure store access in the mix.
…elastic#126801) * Revert endpoint creation validation for ELSER and E5 * Update docs/changelog/126792.yaml * Revert start model deployment being in TransportPutInferenceModelAction --------- Co-authored-by: Elastic Machine <[email protected]>
…ment (elastic#126905) (elastic#126907) Part of for elastic#126897 8.x equivalent of https://github.com/elastic/elasticsearch/pull/126901/files
…#126919) Adds basic coverage for [field capabilities API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-field-caps.html).
Adding basic test coverage for [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html).
Reindex task may not have completed by time status is called, so remove assertion that relies on this. Following assertions are set in the action which starts task, so will be correct whether or not task has completed.
This reverts commit 0a671b2.
…26827) * [8.x] First step optimizing tsdb doc values codec merging. Backporting elastic#125403 to the 8.x branch. The doc values codec iterates a few times over the doc value instance that needs to be written to disk. In case when merging and index sorting is enabled, this is much more expensive, as each time the doc values instance is iterated a merge sorting is performed (in order to get the doc ids of new segment in order of index sorting). There are several reasons why the doc value instance is iterated multiple times: * To compute stats (num values, number of docs with value) required for writing values to disk. * To write bitset that indicate which documents have a value. (indexed disi, jump table) * To write the actual values to disk. * To write the addresses to disk (in case docs have multiple values) This applies for numeric doc values, but also for the ordinals of sorted (set) doc values. This PR addresses solving the first reason why doc value instance needs to be iterated. This is done only when in case of merging and when the segments to be merged with are also of type es87 doc values, codec version is the same and there are no deletes. Note this optimized merged is behind a feature flag for now. * fixed compile errors in benchmark * Fix DocValuesConsumerUtil (elastic#126836) The compatibleWithOptimizedMerge() method doesn't handle codec readers that are wrapped by our source pruning filter codec reader. This change addresses that. Failing to detect this means that the optimized merge will not kick in.
…astic#126923) On x64, we are testing if we support vector capabilities (1 = "basic" = AVX2, 2 = "advanced" = AVX-512) in order to enable and choose a native implementation for some vector functions, using CPUID. However, under some circumstances, this is not sufficient: the OS on which we are running also needs to support AVX/AVX2 etc; basically, it needs to acknowledge it knows about the additional register and that it is able to handle them e.g. in context switches. To do that we need to a) test if the CPU has xsave feature and b) use the xgetbv to test if the OS set it (declaring it supports AVX/AVX2/etc). In most cases this is not needed, as all modern OSes do that, but for some virtualized situations (hypervisors, emulators, etc.) all the component along the chain must support it, and in some cases this is not a given. This PR introduces a change to the x64 version of vec_caps to check for OS support too, and a warning on the Java side in case the CPU supports vector capabilities but those are not enabled at OS level. Tested by passing noxsave to my linux box kernel boot options, and ensuring that the avx flags "disappear" from /proc/cpuinfo, and we fall back to the "no native vector" case. Fixes elastic#126809
…tic#126894) (elastic#126976) The compatibleWithOptimizedMerge() method doesn't the case when doc value fields don't exist in all segments. It is like that for smaller segments not all fields exist. The change addresses that by skipping such doc value producers instead of returning that a field can't be merged using the optimized method.
uncomment blog now that it's live
…lts (elastic#126583) (elastic#126917) * Cancel expired async search task when a remote returns its results (elastic#126583) A while ago we enabled using ccs_minimize_roundtrips in async search. This makes it possible for users of async search to send a single search request per remote cluster, and minimize the impact of network latency. With non minimized roundtrips, we have pretty recurring cancellation checks: as part of the execution, we detect that a task expired whenever each shard comes back with its results. In a scenario where the coord node does not hold data, or only remote data is targeted by an async search, we have much less chance of detecting cancellation if roundtrips are minimized. The local coordinator would do nothing other than waiting for the minimized results from each remote cluster. One scenario where we can check for cancellation is when each cluster comes back with its full set of results. This commit adds such check, plus some testing for async search cancellation with minimized roundtrips. * compile error
…elastic#126850) (elastic#126898) Bump otel-data plugin version as elastic#120952 missed the bump.
…126985) Build jump table (disi) while iterating over SortedNumericDocValues for encoding the values, instead of separately iterating over SortedNumericDocValues just to build the jump table. In case when indexing sorting is active, this requires an additional merge sort. Follow up from elastic#125403
…lastic#126913) The following order of events was possible: - An ILM policy update cleared `cachedSteps` - ILM retrieves the step definition for an index, this populates `cachedSteps` with the outdated policy - The updated policy is put in `lifecyclePolicyMap` Any subsequent cache retrievals will see the old step definition. By clearing `cachedSteps` _after_ we update `lifecyclePolicyMap`, we ensure eventual consistency between the policy and the cache. Fixes elastic#118406 (cherry picked from commit 5383f0f) # Conflicts: # muted-tests.yml
) * ES|QL: make telemetry more strict (elastic#126940) * Fix compile * Fix tests * Fix test
…est (elastic#126930) (elastic#126943) * Adding missing onFailure call * Update docs/changelog/126930.yaml
…126971) * Updating documentation of text_similarity_reranker for 8.x * updating documentation to remove duplicate and redundant wording
These tests had the potential to fail when subsequent requests would hit different nodes with different versions of the cluster state. Only one of these tests failed already, but we fix the other ones proactively to avoid future failures. Fixes elastic#126746 (cherry picked from commit 16070a3) # Conflicts: # muted-tests.yml # test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java # x-pack/plugin/ilm/qa/multi-node/src/javaRestTest/java/org/elasticsearch/xpack/ilm/TimeSeriesDataStreamsIT.java
We addressed the empty top docs issue with elastic#126385 specifically for scenarios where empty top docs don't go through the wire. Yet they may be serialized from data node back to the coord node, in which case they will no longer be equal to Lucene#EMPTY_TOP_DOCS. This commit expands the existing filtering of empty top docs to include also those that did go through serialization. Closes elastic#126742
…#126964) When reading a string value from stdin the keystore add command currently looks directly at stdin. However, stdin may also be consumed while reading the keystore password. This commit changes the add command to use the reader from the termainl instead of looking at stdin directly. closes elastic#98115 Co-authored-by: Elastic Machine <[email protected]>
…c#126862) Updates the SQL grammar to include the selector portion of an index pattern. The index() method has been updated to include selectors in the resulting expression. (cherry picked from commit 299bf44) Co-authored-by: Elastic Machine <[email protected]>
…ing the test factories list (elastic#128211) (elastic#128243) Fixes elastic#128092 The error happens inside an ArrayList.add(), and it's a concurrency-related one. The stacktrace is quite explicative: ``` java.lang.ArrayIndexOutOfBoundsException: Index 34 out of bounds for length 33 at java.util.ArrayList.add(ArrayList.java:485) at org.elasticsearch.compute.test.ComputeTestCase.blockFactory(ComputeTestCase.java:62) at org.elasticsearch.compute.aggregation.GroupingAggregatorFunctionTestCase$4$1$1.addBlock(GroupingAggregatorFunctionTestCase.java:659) ``` We were creating new blockFactories() inside the aggregator, which was being called concurrently with different drivers. As `blockFactory()` adds it to a static ArrayList, it was sometimes throwing. The bug only affects tests. Potentially, all grouping aggregators tests.
…8223) (elastic#128248) * Use package to suppress warning for entitlement self-test * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <[email protected]>
…stic#128231) * ESQL: Avoid unintended attribute removal (elastic#127563) --------- Co-authored-by: Andrei Stefan <[email protected]> * Checkstyle * Checkstyle again * Slightly change the test because 8.19 has fewer indices in the index pattern used (9.x also has host_inventory index). --------- Co-authored-by: kanoshiou <[email protected]>
This commit continues the work of removing SecurityManager related code by removing doPrivileged calls from Elasticsearch plugins.
Fixed typo (cherry picked from commit 5699763) Co-authored-by: jamie-wilson88 <[email protected]>
* Add temporary LegacyActionRequest (elastic#128107) In order to remove ActionType, ActionRequest will become strongly typed, referring to the ActionResponse type. As a precursor to that, this commit adds a LegacyActionRequest which all existing ActionRequest implementations now inherit from. This will allow adding the ActionResponse type to ActionRequest in a future commit without modifying every implementation at once. * fix oops * use list directly * more test compile
…c#128266) * Add NamedWriteable for QueryRule rank doc * Update test * Update docs/changelog/128153.yaml * Add multi cluster test for query rules * Commenting out code - explicitly trying to spur a test failure * [CI] Auto commit changes from spotless * Streamline test for multi cluster * Revert changes to try to break test * Fix compile error --------- Co-authored-by: elasticsearchmachine <[email protected]> (cherry picked from commit b335c1a) # Conflicts: # x-pack/plugin/ent-search/src/yamlRestTest/resources/rest-api-spec/test/entsearch/rules/40_rule_query_search.yml
…ts (elastic#127486) (elastic#128280) Generates a test file with the following information and format: { "component": "<component name>", "locations": [ { "representative_class": <class name with package>, "module": "<module name>" }, ... ] } For painless: { "component": "lang-painless", "locations": [ { "representative_class": "org/objectweb/asm/tree/analysis/Analyzer.class", "module": "org.objectweb.asm.tree.analysis" }, ... ] } Then it copies the following files into the jar for consumption by unit tests: * META-INF/plugin-test-build-info.json * META-INF/es-plugins/<plugin name>/plugin-descriptor.properties * META-INF/es-plugins/<plugin name>/entitlement-policy.yaml For server, the files in the jar become the following: * META-INF/server-test-build-info.json This should provide enough information for BootstrapForTesting to be able to build a mapping of caller class to policy file using the class file to look up the jar or directory within the class path and then associating that with it's specified module and finally using the specified module to look up the appropriate entitlement policy. caller class -> specified module -> entitlement policy --------- Co-authored-by: Patrick Doyle <[email protected]>
Today there are various mechanisms to prevent writes to readonly repositories, but they are scattered across the snapshot codebase and do not obviously prevent writes in all possible circumstances; it'd be easy to add a new operation on a repository that does not check the readonly flag in quite the right way. This commit adds much tighter checks which cannot be circumvented: - Do not allow to start an update of the root `index-N` blob if the repository is marked as readonly in the cluster state. - Conversely, do not allow the readonly flag to be set if an update of the root `index-N` blob is in progress. - Establish the invariant that we never create a `SnapshotsInProgress$Entry`, `SnapshotDeletionsInProgress$Entry`, or `RepositoryCleanupInProgress$Entry` if the repository is marked as readonly in the cluster state. Closes elastic#93575 Backport of elastic#127964 to `8.19`
…solver) (elastic#127719) (elastic#128301) This PR introduces a test-specific ScopeResolver to use with PolicyManager for checking entitlements within test code running in a test runner (unit tests and integ tests, where code is running withing the same JVM). The information for resolving component and module names is derived from the file created in elastic#127486
…lastic#128213) (elastic#128312) This change updates the code to always create SourceProvider instances via MappingLookup, avoiding direct exposure to the underlying source format (synthetic or stored). It also aligns source filtering behaviour between SourceProvider and SourceLoader, ensuring consistent application of filters. This change is needed to enable source filtering to occur earlier in the fetch phase, for example, when constructing a synthetic source.
…elastic#128332) * Using correct origin for inference action * Update docs/changelog/128327.yaml * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <[email protected]> (cherry picked from commit 19e18a9) # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/inference/InferenceRunner.java
Skip some tests that expect a warning for encountering multivalued fields against versions that don't emit this warnings. Closes elastic#128224
(cherry picked from commit 2f01d17)
…#128349) Our APMTracer doesn't like nulls - this is a sensible thing, as APM in general does not allow nulls (it only allows a precise set of types). This PR changes the attribute to a sentinel "" in place of null values. It also makes a small change to APMTracer to give a better error message in case of null values in attributes.
…eIT test {yaml=search.vectors/41_knn_search_byte_quantized/Test index configured rescore vector score consistency} elastic#128353
* Add Hugging Face Rerank support * Address comments * Add transport version * Add transport version * Add to inference service and crud IT rerank tests * Refactor slightly / error message * correct 'testGetConfiguration' test case * apply suggestions * fix tests * apply suggestions * [CI] Auto commit changes from spotless * add changelog information --------- Co-authored-by: elasticsearchmachine <[email protected]>
elasticsearchmachine
added
v9.1.0
needs:triage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
64 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
gradle check?