feat: [google-cloud-discoveryengine] Added ranking_expression_backed and rank_signals fields related to the Custom Ranking feature (googleapis#14393)

BEGIN_COMMIT_OVERRIDE
feat: Added `ranking_expression_backed` and `rank_signals` fields
related to the Custom Ranking feature
docs: A comment for field `ranking_expression` in messages
`.google.cloud.discoveryengine.v1alpha.SearchRequest` and
`.google.cloud.discoveryengine.v1beta.SearchRequest` is changed to
support the Custom Ranking use case
END_COMMIT_OVERRIDE

- [ ] Regenerate this pull request now.

docs: A comment for field `ranking_expression` in messages
`.google.cloud.discoveryengine.v1alpha.SearchRequest` and
`.google.cloud.discoveryengine.v1beta.SearchRequest` is changed to
support the Custom Ranking use case

PiperOrigin-RevId: 805306845

Source-Link:
googleapis/googleapis@0ef98bc

Source-Link:
https://github.com/googleapis/googleapis-gen/commit/79a868820f3426367466456fc7e134a87b7c1ef1
Copy-Tag:
eyJwIjoicGFja2FnZXMvZ29vZ2xlLWNsb3VkLWRpc2NvdmVyeWVuZ2luZS8uT3dsQm90LnlhbWwiLCJoIjoiNzlhODY4ODIwZjM0MjYzNjc0NjY0NTZmYzdlMTM0YTg3YjdjMWVmMSJ9

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Author: gcf-owl-bot[bot]

packages/google-cloud-discoveryengine/google/cloud/discoveryengine_v1/types/search_service.py

@@ -301,6 +301,112 @@ class SearchRequest(proto.Message):
         relevance_score_spec (google.cloud.discoveryengine_v1.types.SearchRequest.RelevanceScoreSpec):
             Optional. The specification for returning the
             relevance score.
+        ranking_expression (str):
+            The ranking expression controls the customized ranking on
+            retrieval documents. This overrides
+            [ServingConfig.ranking_expression][google.cloud.discoveryengine.v1.ServingConfig.ranking_expression].
+            The syntax and supported features depend on the
+            ``ranking_expression_backend`` value. If
+            ``ranking_expression_backend`` is not provided, it defaults
+            to ``RANK_BY_EMBEDDING``.
+
+            If
+            [ranking_expression_backend][google.cloud.discoveryengine.v1.SearchRequest.ranking_expression_backend]
+            is not provided or set to ``RANK_BY_EMBEDDING``, it should
+            be a single function or multiple functions that are joined
+            by "+".
+
+            - ranking_expression = function, { " + ", function };
+
+            Supported functions:
+
+            - double \* relevance_score
+            - double \* dotProduct(embedding_field_path)
+
+            Function variables:
+
+            - ``relevance_score``: pre-defined keywords, used for
+              measure relevance between query and document.
+            - ``embedding_field_path``: the document embedding field
+              used with query embedding vector.
+            - ``dotProduct``: embedding function between
+              ``embedding_field_path`` and query embedding vector.
+
+            Example ranking expression:
+
+            ::
+
+               If document has an embedding field doc_embedding, the ranking expression
+               could be `0.5 * relevance_score + 0.3 * dotProduct(doc_embedding)`.
+
+            If
+            [ranking_expression_backend][google.cloud.discoveryengine.v1.SearchRequest.ranking_expression_backend]
+            is set to ``RANK_BY_FORMULA``, the following expression
+            types (and combinations of those chained using + or
+
+            - operators) are supported:
+
+              - ``double``
+              - ``signal``
+              - ``log(signal)``
+              - ``exp(signal)``
+              - ``rr(signal, double > 0)`` -- reciprocal rank
+                transformation with second argument being a denominator
+                constant.
+              - ``is_nan(signal)`` -- returns 0 if signal is NaN, 1
+                otherwise.
+              - ``fill_nan(signal1, signal2 | double)`` -- if signal1 is
+                NaN, returns signal2 \| double, else returns signal1.
+
+              Here are a few examples of ranking formulas that use the
+              supported ranking expression types:
+
+              - ``0.2 * semantic_similarity_score + 0.8 * log(keyword_similarity_score)``
+                -- mostly rank by the logarithm of
+                ``keyword_similarity_score`` with slight
+                ``semantic_smilarity_score`` adjustment.
+              - ``0.2 * exp(fill_nan(semantic_similarity_score, 0)) + 0.3 * is_nan(keyword_similarity_score)``
+                -- rank by the exponent of ``semantic_similarity_score``
+                filling the value with 0 if it's NaN, also add constant
+                0.3 adjustment to the final score if
+                ``semantic_similarity_score`` is NaN.
+              - ``0.2 * rr(semantic_similarity_score, 16) + 0.8 * rr(keyword_similarity_score, 16)``
+                -- mostly rank by the reciprocal rank of
+                ``keyword_similarity_score`` with slight adjustment of
+                reciprocal rank of ``semantic_smilarity_score``.
+
+            The following signals are supported:
+
+            - ``semantic_similarity_score``: semantic similarity
+              adjustment that is calculated using the embeddings
+              generated by a proprietary Google model. This score
+              determines how semantically similar a search query is to a
+              document.
+            - ``keyword_similarity_score``: keyword match adjustment
+              uses the Best Match 25 (BM25) ranking function. This score
+              is calculated using a probabilistic model to estimate the
+              probability that a document is relevant to a given query.
+            - ``relevance_score``: semantic relevance adjustment that
+              uses a proprietary Google model to determine the meaning
+              and intent behind a user's query in context with the
+              content in the documents.
+            - ``pctr_rank``: predicted conversion rate adjustment as a
+              rank use predicted Click-through rate (pCTR) to gauge the
+              relevance and attractiveness of a search result from a
+              user's perspective. A higher pCTR suggests that the result
+              is more likely to satisfy the user's query and intent,
+              making it a valuable signal for ranking.
+            - ``freshness_rank``: freshness adjustment as a rank
+            - ``document_age``: The time in hours elapsed since the
+              document was last updated, a floating-point number (e.g.,
+              0.25 means 15 minutes).
+            - ``topicality_rank``: topicality adjustment as a rank. Uses
+              proprietary Google model to determine the keyword-based
+              overlap between the query and the document.
+            - ``base_rank``: the default rank of the result
+        ranking_expression_backend (google.cloud.discoveryengine_v1.types.SearchRequest.RankingExpressionBackend):
+            The backend to use for the ranking expression
+            evaluation.
     """
 
     class RelevanceThreshold(proto.Enum):
@@ -327,6 +433,23 @@ class RelevanceThreshold(proto.Enum):
         MEDIUM = 3
         HIGH = 4
 
+    class RankingExpressionBackend(proto.Enum):
+        r"""The backend to use for the ranking expression evaluation.
+
+        Values:
+            RANKING_EXPRESSION_BACKEND_UNSPECIFIED (0):
+                Default option for unspecified/unknown
+                values.
+            RANK_BY_EMBEDDING (3):
+                Ranking by custom embedding model, the
+                default way to evaluate the ranking expression.
+            RANK_BY_FORMULA (4):
+                Ranking by custom formula.
+        """
+        RANKING_EXPRESSION_BACKEND_UNSPECIFIED = 0
+        RANK_BY_EMBEDDING = 3
+        RANK_BY_FORMULA = 4
+
     class ImageQuery(proto.Message):
         r"""Specifies the image query input.
 
@@ -1540,6 +1663,15 @@ class RelevanceScoreSpec(proto.Message):
         number=52,
         message=RelevanceScoreSpec,
     )
+    ranking_expression: str = proto.Field(
+        proto.STRING,
+        number=26,
+    )
+    ranking_expression_backend: RankingExpressionBackend = proto.Field(
+        proto.ENUM,
+        number=53,
+        enum=RankingExpressionBackend,
+    )
 
 
 class SearchResponse(proto.Message):
@@ -1620,8 +1752,119 @@ class SearchResult(proto.Message):
             model_scores (MutableMapping[str, google.cloud.discoveryengine_v1.types.DoubleList]):
                 Output only. Google provided available
                 scores.
+            rank_signals (google.cloud.discoveryengine_v1.types.SearchResponse.SearchResult.RankSignals):
+                A set of ranking signals associated with the
+                result.
         """
 
+        class RankSignals(proto.Message):
+            r"""A set of ranking signals.
+
+            .. _oneof: https://proto-plus-python.readthedocs.io/en/stable/fields.html#oneofs-mutually-exclusive-fields
+
+            Attributes:
+                keyword_similarity_score (float):
+                    Keyword matching adjustment.
+
+                    This field is a member of `oneof`_ ``_keyword_similarity_score``.
+                relevance_score (float):
+                    Semantic relevance adjustment.
+
+                    This field is a member of `oneof`_ ``_relevance_score``.
+                semantic_similarity_score (float):
+                    Semantic similarity adjustment.
+
+                    This field is a member of `oneof`_ ``_semantic_similarity_score``.
+                pctr_rank (float):
+                    Predicted conversion rate adjustment as a
+                    rank.
+
+                    This field is a member of `oneof`_ ``_pctr_rank``.
+                topicality_rank (float):
+                    Topicality adjustment as a rank.
+
+                    This field is a member of `oneof`_ ``_topicality_rank``.
+                document_age (float):
+                    Age of the document in hours.
+
+                    This field is a member of `oneof`_ ``_document_age``.
+                boosting_factor (float):
+                    Combined custom boosts for a doc.
+
+                    This field is a member of `oneof`_ ``_boosting_factor``.
+                default_rank (float):
+                    The default rank of the result.
+                custom_signals (MutableSequence[google.cloud.discoveryengine_v1.types.SearchResponse.SearchResult.RankSignals.CustomSignal]):
+                    A list of custom clearbox signals.
+            """
+
+            class CustomSignal(proto.Message):
+                r"""Custom clearbox signal represented by name and value pair.
+
+                Attributes:
+                    name (str):
+                        Name of the signal.
+                    value (float):
+                        Float value representing the ranking signal
+                        (e.g. 1.25 for BM25).
+                """
+
+                name: str = proto.Field(
+                    proto.STRING,
+                    number=1,
+                )
+                value: float = proto.Field(
+                    proto.FLOAT,
+                    number=2,
+                )
+
+            keyword_similarity_score: float = proto.Field(
+                proto.FLOAT,
+                number=1,
+                optional=True,
+            )
+            relevance_score: float = proto.Field(
+                proto.FLOAT,
+                number=2,
+                optional=True,
+            )
+            semantic_similarity_score: float = proto.Field(
+                proto.FLOAT,
+                number=3,
+                optional=True,
+            )
+            pctr_rank: float = proto.Field(
+                proto.FLOAT,
+                number=4,
+                optional=True,
+            )
+            topicality_rank: float = proto.Field(
+                proto.FLOAT,
+                number=6,
+                optional=True,
+            )
+            document_age: float = proto.Field(
+                proto.FLOAT,
+                number=7,
+                optional=True,
+            )
+            boosting_factor: float = proto.Field(
+                proto.FLOAT,
+                number=8,
+                optional=True,
+            )
+            default_rank: float = proto.Field(
+                proto.FLOAT,
+                number=32,
+            )
+            custom_signals: MutableSequence[
+                "SearchResponse.SearchResult.RankSignals.CustomSignal"
+            ] = proto.RepeatedField(
+                proto.MESSAGE,
+                number=33,
+                message="SearchResponse.SearchResult.RankSignals.CustomSignal",
+            )
+
         id: str = proto.Field(
             proto.STRING,
             number=1,
@@ -1642,6 +1885,11 @@ class SearchResult(proto.Message):
             number=4,
             message=common.DoubleList,
         )
+        rank_signals: "SearchResponse.SearchResult.RankSignals" = proto.Field(
+            proto.MESSAGE,
+            number=7,
+            message="SearchResponse.SearchResult.RankSignals",
+        )
 
     class Facet(proto.Message):
         r"""A facet result.