Skip to content

Commit b856451

Browse files
gaobinlonggaobinlong
authored
Field collapsing supports search_after (#19261)
* Field collapsing supports search_after Signed-off-by: Binlong Gao <[email protected]> * Modify change log Signed-off-by: Binlong Gao <[email protected]> * Avoid changing public APIs Signed-off-by: Binlong Gao <[email protected]> * Revert change for the existing method Signed-off-by: Binlong Gao <[email protected]> * Format change log Signed-off-by: Binlong Gao <[email protected]> * Fix some issues Signed-off-by: Binlong Gao <[email protected]> * Remove duplicate code Signed-off-by: Binlong Gao <[email protected]> * Fix format issue Signed-off-by: Binlong Gao <[email protected]> --------- Signed-off-by: Binlong Gao <[email protected]>
1 parent 10f8c8e commit b856451

File tree

7 files changed

+686
-20
lines changed

7 files changed

+686
-20
lines changed

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
1515
- Add a dynamic setting to change skip_cache_factor and min_frequency for querycache ([#18351](https://github.com/opensearch-project/OpenSearch/issues/18351))
1616
- Add overload constructor for Translog to accept Channel Factory as a parameter ([#18918](https://github.com/opensearch-project/OpenSearch/pull/18918))
1717
- Add subdirectory-aware store module with recovery support ([#19132](https://github.com/opensearch-project/OpenSearch/pull/19132))
18+
- Field collapsing supports search_after ([#19261](https://github.com/opensearch-project/OpenSearch/pull/19261))
1819
- Add a dynamic cluster setting to control the enablement of the merged segment warmer ([#18929](https://github.com/opensearch-project/OpenSearch/pull/18929))
1920
- Publish transport-grpc-spi exposing QueryBuilderProtoConverter and QueryBuilderProtoConverterRegistry ([#18949](https://github.com/opensearch-project/OpenSearch/pull/18949))
2021
- Support system generated search pipeline. ([#19128](https://github.com/opensearch-project/OpenSearch/pull/19128))

‎rest-api-spec/src/main/resources/rest-api-spec/test/search/110_field_collapsing.yml‎

Lines changed: 331 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -245,20 +245,6 @@ setup:
245245
body:
246246
collapse: { field: numeric_group }
247247

248-
---
249-
"field collapsing and search_after":
250-
251-
- do:
252-
catch: /cannot use \`collapse\` in conjunction with \`search_after\`/
253-
search:
254-
allow_partial_search_results: false
255-
rest_total_hits_as_int: true
256-
index: test
257-
body:
258-
collapse: { field: numeric_group }
259-
search_after: [6]
260-
sort: [{ sort: desc }]
261-
262248
---
263249
"field collapsing and rescore":
264250

@@ -610,3 +596,334 @@ setup:
610596
- match: { hits.hits.0._source.marker: 'doc1' }
611597
- match: { hits.hits.1._id: '2' }
612598
- match: { hits.hits.1._source.marker: 'doc2' }
599+
600+
601+
---
602+
"field collapsing with search_after - basic functionality":
603+
- skip:
604+
version: " - 3.2.99"
605+
reason: Introduced in 3.3.0
606+
- do:
607+
search:
608+
rest_total_hits_as_int: true
609+
index: test
610+
body:
611+
collapse: { field: numeric_group }
612+
sort: [{ numeric_group: asc }]
613+
size: 10
614+
615+
- match: {hits.total: 6 }
616+
- length: {hits.hits: 3 }
617+
- match: {hits.hits.0.fields.numeric_group: [1] }
618+
- match: {hits.hits.1.fields.numeric_group: [3] }
619+
- match: {hits.hits.2.fields.numeric_group: [25] }
620+
621+
- do:
622+
catch: /collapse field and sort field must be the same when use `collapse` in conjunction with `search_after`/
623+
search:
624+
rest_total_hits_as_int: true
625+
index: test
626+
body:
627+
collapse: { field: numeric_group }
628+
sort: [{ sort: desc }]
629+
search_after: [10]
630+
- do:
631+
catch: /collapse field and sort field must be the same when use `collapse` in conjunction with `search_after`/
632+
search:
633+
rest_total_hits_as_int: true
634+
index: test
635+
body:
636+
collapse: { field: numeric_group }
637+
sort: [{ numeric_group: desc }, { sort: asc }]
638+
search_after: [25, 10]
639+
640+
# Test asc, first page
641+
- do:
642+
search:
643+
rest_total_hits_as_int: true
644+
index: test
645+
body:
646+
collapse: { field: numeric_group }
647+
sort: [{ numeric_group: asc }]
648+
search_after: [1]
649+
size: 10
650+
651+
- match: {hits.total: 6 }
652+
- length: {hits.hits: 2 }
653+
- match: {hits.hits.0.fields.numeric_group: [3] }
654+
- match: {hits.hits.1.fields.numeric_group: [25] }
655+
656+
# Test asc, second page
657+
- do:
658+
search:
659+
rest_total_hits_as_int: true
660+
index: test
661+
body:
662+
collapse: { field: numeric_group }
663+
sort: [{ numeric_group: asc }]
664+
search_after: [3]
665+
size: 2
666+
667+
- match: {hits.total: 6 }
668+
- length: {hits.hits: 1 }
669+
- match: {hits.hits.0.fields.numeric_group: [25] }
670+
671+
# Test asc, no result
672+
- do:
673+
search:
674+
rest_total_hits_as_int: true
675+
index: test
676+
body:
677+
collapse: { field: numeric_group }
678+
sort: [{ numeric_group: asc }]
679+
search_after: [999]
680+
size: 10
681+
682+
- match: {hits.total: 6 }
683+
- length: {hits.hits: 0 }
684+
685+
# Test desc, first page
686+
- do:
687+
search:
688+
rest_total_hits_as_int: true
689+
index: test
690+
body:
691+
collapse: { field: numeric_group }
692+
sort: [{ numeric_group: desc }]
693+
size: 1
694+
695+
- match: {hits.total: 6 }
696+
- length: {hits.hits: 1 }
697+
- match: {hits.hits.0.fields.numeric_group: [25] }
698+
- set: { hits.hits.0.sort.0: last_sort_value }
699+
700+
# Test desc, second page
701+
- do:
702+
search:
703+
rest_total_hits_as_int: true
704+
index: test
705+
body:
706+
collapse: { field: numeric_group }
707+
sort: [{ numeric_group: desc }]
708+
search_after: [$last_sort_value]
709+
size: 1
710+
711+
- match: {hits.total: 6 }
712+
- length: {hits.hits: 1 }
713+
- match: {hits.hits.0.fields.numeric_group: [3] }
714+
- set: { hits.hits.0.sort.0: last_sort_value }
715+
716+
# Test desc, third page
717+
- do:
718+
search:
719+
rest_total_hits_as_int: true
720+
index: test
721+
body:
722+
collapse: { field: numeric_group }
723+
sort: [{ numeric_group: desc }]
724+
search_after: [$last_sort_value]
725+
size: 1
726+
727+
- match: {hits.total: 6 }
728+
- length: {hits.hits: 1 }
729+
- match: {hits.hits.0.fields.numeric_group: [1] }
730+
- set: { hits.hits.0.sort.0: last_sort_value }
731+
732+
# Test desc, no result
733+
- do:
734+
search:
735+
rest_total_hits_as_int: true
736+
index: test
737+
body:
738+
collapse: { field: numeric_group }
739+
sort: [{ numeric_group: desc }]
740+
search_after: [$last_sort_value]
741+
size: 1
742+
743+
- match: {hits.total: 6 }
744+
- length: {hits.hits: 0 }
745+
746+
# test on keyword field
747+
- do:
748+
search:
749+
rest_total_hits_as_int: true
750+
index: test
751+
body:
752+
collapse: { field: tag }
753+
sort: [{ tag: asc }]
754+
size: 1
755+
756+
- match: {hits.total: 6 }
757+
- length: {hits.hits: 1 }
758+
- match: {hits.hits.0.fields.tag: ["A"] }
759+
760+
# Search after "A"
761+
- do:
762+
search:
763+
rest_total_hits_as_int: true
764+
index: test
765+
body:
766+
collapse: { field: tag }
767+
sort: [{ tag: asc }]
768+
search_after: ["A"]
769+
size: 1
770+
771+
---
772+
"field collapsing with search_after - concurrent segment search enabled":
773+
- skip:
774+
version: " - 3.2.99"
775+
reason: Introduced in 3.3.0
776+
- do:
777+
indices.put_settings:
778+
index: test
779+
body:
780+
index.search.concurrent_segment_search.mode: 'all'
781+
782+
# Test asc, first page
783+
- do:
784+
search:
785+
rest_total_hits_as_int: true
786+
index: test
787+
body:
788+
collapse: { field: numeric_group }
789+
sort: [{ numeric_group: asc }]
790+
search_after: [1]
791+
size: 10
792+
793+
- match: {hits.total: 6 }
794+
- length: {hits.hits: 2 }
795+
- match: {hits.hits.0.fields.numeric_group: [3] }
796+
- match: {hits.hits.1.fields.numeric_group: [25] }
797+
798+
# Test asc, second page
799+
- do:
800+
search:
801+
rest_total_hits_as_int: true
802+
index: test
803+
body:
804+
collapse: { field: numeric_group }
805+
sort: [{ numeric_group: asc }]
806+
search_after: [3]
807+
size: 2
808+
809+
- match: {hits.total: 6 }
810+
- length: {hits.hits: 1 }
811+
- match: {hits.hits.0.fields.numeric_group: [25] }
812+
813+
# Test asc, no result
814+
- do:
815+
search:
816+
rest_total_hits_as_int: true
817+
index: test
818+
body:
819+
collapse: { field: numeric_group }
820+
sort: [{ numeric_group: asc }]
821+
search_after: [999]
822+
size: 10
823+
824+
- match: {hits.total: 6 }
825+
- length: {hits.hits: 0 }
826+
827+
# Test desc, first page
828+
- do:
829+
search:
830+
rest_total_hits_as_int: true
831+
index: test
832+
body:
833+
collapse: { field: numeric_group }
834+
sort: [{ numeric_group: desc }]
835+
size: 1
836+
837+
- match: {hits.total: 6 }
838+
- length: {hits.hits: 1 }
839+
- match: {hits.hits.0.fields.numeric_group: [25] }
840+
- set: { hits.hits.0.sort.0: last_sort_value }
841+
842+
# Test desc, second page
843+
- do:
844+
search:
845+
rest_total_hits_as_int: true
846+
index: test
847+
body:
848+
collapse: { field: numeric_group }
849+
sort: [{ numeric_group: desc }]
850+
search_after: [$last_sort_value]
851+
size: 1
852+
853+
- match: {hits.total: 6 }
854+
- length: {hits.hits: 1 }
855+
- match: {hits.hits.0.fields.numeric_group: [3] }
856+
- set: { hits.hits.0.sort.0: last_sort_value }
857+
858+
# Test desc, third page
859+
- do:
860+
search:
861+
rest_total_hits_as_int: true
862+
index: test
863+
body:
864+
collapse: { field: numeric_group }
865+
sort: [{ numeric_group: desc }]
866+
search_after: [$last_sort_value]
867+
size: 1
868+
869+
- match: {hits.total: 6 }
870+
- length: {hits.hits: 1 }
871+
- match: {hits.hits.0.fields.numeric_group: [1] }
872+
- set: { hits.hits.0.sort.0: last_sort_value }
873+
874+
# Test desc, no result
875+
- do:
876+
search:
877+
rest_total_hits_as_int: true
878+
index: test
879+
body:
880+
collapse: { field: numeric_group }
881+
sort: [{ numeric_group: desc }]
882+
search_after: [$last_sort_value]
883+
size: 1
884+
885+
- match: {hits.total: 6 }
886+
- length: {hits.hits: 0 }
887+
888+
# test on keyword field
889+
- do:
890+
search:
891+
rest_total_hits_as_int: true
892+
index: test
893+
body:
894+
collapse: { field: tag }
895+
sort: [{ tag: asc }]
896+
size: 1
897+
898+
- match: {hits.total: 6 }
899+
- length: {hits.hits: 1 }
900+
- match: {hits.hits.0.fields.tag: ["A"] }
901+
902+
# Search after "A"
903+
- do:
904+
search:
905+
rest_total_hits_as_int: true
906+
index: test
907+
body:
908+
collapse: { field: tag }
909+
sort: [{ tag: asc }]
910+
search_after: ["A"]
911+
size: 1
912+
913+
- match: {hits.total: 6 }
914+
- length: {hits.hits: 1 }
915+
- match: {hits.hits.0.fields.tag: ["B"] }
916+
917+
# Search after "B"
918+
- do:
919+
search:
920+
rest_total_hits_as_int: true
921+
index: test
922+
body:
923+
collapse: { field: tag }
924+
sort: [{ tag: asc }]
925+
search_after: ["B"]
926+
size: 1
927+
928+
- match: {hits.total: 6 }
929+
- length: {hits.hits: 0 }

0 commit comments

Comments
 (0)