Edit vector store fields.

[d3buggerdan]

metadata_fields = {
"date": ("date", MetadataIndexFieldType.STRING),
"userid": ("userid", MetadataIndexFieldType.STRING),
"contractid": ("contractid", MetadataIndexFieldType.STRING),
"title": ("title", MetadataIndexFieldType.STRING)
}

Define the vector store

vector_store = AzureAISearchVectorStore(
search_or_index_client=index_client,
filterable_metadata_field_keys=metadata_fields,
index_name=index_name,
index_management=IndexManagement.CREATE_IF_NOT_EXISTS,
id_field_key="id",
chunk_field_key="chunk",
embedding_field_key="embedding",
embedding_dimensionality=1536,
metadata_string_field_key="metadata",
doc_id_field_key="doc_id",
language_analyzer="en.lucene",
vector_algorithm_type="exhaustiveKnn",
)

I want the doc_id, date, userid, contractid, and title to be sortable and searchable as well, right now they are only retrievable and filterable

@dosu

[dosubot[bot]]

To make the doc_id, date, userid, contractid, and title fields sortable and searchable, you need to update the configuration of these fields in the fields list. Specifically, you should use the SearchableField for fields that need to be searchable and sortable.

Here is the updated configuration:

logger.info(f"Configuring {index_name} fields for Azure AI Search")
fields = [
    SimpleField(name=self._field_mapping["id"], type="Edm.String", key=True),
    SearchableField(
        name=self._field_mapping["chunk"],
        type="Edm.String",
        analyzer_name=self._language_analyzer,
    ),
    SearchField(
        name=self._field_mapping["embedding"],
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
        searchable=True,
        vector_search_dimensions=self._embedding_dimensionality,
        vector_search_profile_name=self._vector_profile_name,
    ),
    SimpleField(name=self._field_mapping["metadata"], type="Edm.String"),
    SearchableField(
        name=self._field_mapping["doc_id"], type="Edm.String", filterable=True, sortable=True
    ),
    SearchableField(
        name=self._field_mapping["date"], type="Edm.String", filterable=True, sortable=True
    ),
    SearchableField(
        name=self._field_mapping["userid"], type="Edm.String", filterable=True, sortable=True
    ),
    SearchableField(
        name=self._field_mapping["contractid"], type="Edm.String", filterable=True, sortable=True
    ),
    SearchableField(
        name=self._field_mapping["title"], type="Edm.String", filterable=True, sortable=True
    ),
]
logger.info(f"Configuring {index_name} metadata fields")
metadata_index_fields = self._create_metadata_index_fields()
fields.extend(metadata_index_fields)
logger.info(f"Configuring {index_name} vector search")
# Configure the vector search algorithms and profiles
vector_search = VectorSearch(
    algorithms=[
        HnswAlgorithmConfiguration(
            name="myHnsw",
            kind=VectorSearchAlgorithmKind.HNSW,
            parameters=HnswParameters(
                m=4,
                ef_construction=400,
                ef_search=500,
                metric=VectorSearchAlgorithmMetric.COSINE,
            ),
        ),
        ExhaustiveKnnAlgorithmConfiguration(
            name="myExhaustiveKnn",
            kind=VectorSearchAlgorithmKind.EXHAUSTIVE_KNN,
            parameters=ExhaustiveKnnParameters(
                metric=VectorSearchAlgorithmMetric.COSINE,
            ),
        ),
    ],
    profiles=[
        VectorSearchProfile(
            name="myHnswProfile",
            algorithm_configuration_name="myHnsw",
        ),
        VectorSearchProfile(
            name="myExhaustiveKnnProfile",
            algorithm_configuration_name="myExhaustiveKnn",
        ),
    ],
)
logger.info(f"Configuring {index_name} semantic search")
semantic_config = SemanticConfiguration(
    name="mySemanticConfig",
    prioritized_fields=SemanticPrioritizedFields(
        content_fields=[SemanticField(field_name=self._field_mapping["chunk"])],
    ),
)

semantic_search = SemanticSearch(configurations=[semantic_config])

index = SearchIndex(
    name=index_name,
    fields=fields,
    vector_search=vector_search,
    semantic_search=semantic_search,
)
logger.debug(f"Creating {index_name} search index")
self._index_client.create_index(index)

This configuration ensures that the doc_id, date, userid, contractid, and title fields are both searchable and sortable ^[1][2].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To implement a decomposable query with your setup, you need to use the Query Transforms module to decompose the query into sub-queries. Here's how you can integrate it into your existing code:

1. Define the Query Transform:
   Use one of the available query transforms, such as DecomposeQueryTransform, to handle the decomposition of queries.

2. Integrate the Query Transform with the Retriever:
   Update your retriever to include the query transform.

3. Configure the Query Engine:
   Ensure the RetrieverQueryEngine is set up to handle the decomposed queries.

Here's an example of how to do this:

from llama_index.core.indices.query.query_transform.base import DecomposeQueryTransform
from llama_index.core.query_engine.retriever_query_engine import RetrieverQueryEngine
from llama_index.core.response_synthesizers.factory import get_response_synthesizer
from llama_index.core.vector_stores.types import VectorStoreQueryMode

# Define the query transform
query_transform = DecomposeQueryTransform()

# Create the retriever with the query transform
default_retriever = index.as_retriever(
    vector_store_query_mode=VectorStoreQueryMode.HYBRID,
    similarity_top_k=10,
    query_transform=query_transform
)

# Create the response synthesizer
response_synthesizer = get_response_synthesizer(
    response_mode="tree_summarize",
    streaming=True,
)

# Create the query engine
query_engine = RetrieverQueryEngine(
    retriever=default_retriever,
    response_synthesizer=response_synthesizer,
)

# Perform the query
response = query_engine.query("Your query here")

In this setup:

• DecomposeQueryTransform is used to decompose the query into sub-queries.
• The default_retriever is configured with the query_transform.
• The RetrieverQueryEngine handles the decomposed queries and synthesizes the final response.

This approach ensures that your queries are decomposed into manageable parts, improving the accuracy and completeness of the responses ^[1][2].

To continue talking to Dosu, mention @dosu.

[d3buggerdan]

@dosu can i print out what it decomposed the query to?

[dosubot[bot]]

Yes, you can print out the sub-queries that the DecomposeQueryTransform decomposed the original query into by setting the verbose parameter to True when initializing the DecomposeQueryTransform object. This will enable the printing of the current and new queries during the transformation process.

Here's how you can do it:

from llama_index.core.indices.query.query_transform.base import DecomposeQueryTransform
from llama_index.core.query_engine.retriever_query_engine import RetrieverQueryEngine
from llama_index.core.response_synthesizers.factory import get_response_synthesizer
from llama_index.core.vector_stores.types import VectorStoreQueryMode

# Initialize the DecomposeQueryTransform with verbose=True
query_transform = DecomposeQueryTransform(verbose=True)

# Create the retriever with the query transform
default_retriever = index.as_retriever(
    vector_store_query_mode=VectorStoreQueryMode.HYBRID,
    similarity_top_k=10,
    query_transform=query_transform
)

# Create the response synthesizer
response_synthesizer = get_response_synthesizer(
    response_mode="tree_summarize",
    streaming=True,
)

# Create the query engine
query_engine = RetrieverQueryEngine(
    retriever=default_retriever,
    response_synthesizer=response_synthesizer,
)

# Perform the query
response = query_engine.query("Your query here")

With verbose=True, the DecomposeQueryTransform will print the current and new queries during the _run method execution:

if self.verbose:
    print_text(f"> Current query: {query_str}\n", color="yellow")
    print_text(f"> New query: {new_query_str}\n", color="pink")

This will output the sub-queries to the console, allowing you to see how the original query was decomposed ^[1][2].

To continue talking to Dosu, mention @dosu.

[d3buggerdan]

@dosu i set verbose to true and it does not show the sub-queries

[dosubot[bot]]

To troubleshoot the issue of sub-queries not being printed despite setting verbose=True in the DecomposeQueryTransform, ensure that the verbose attribute is correctly set and that the print_text function is functioning as expected.

Here is the relevant code snippet from the DecomposeQueryTransform class:

class DecomposeQueryTransform(BaseQueryTransform):
    ...
    def __init__(
        self,
        llm: Optional[LLMPredictorType] = None,
        decompose_query_prompt: Optional[DecomposeQueryTransformPrompt] = None,
        verbose: bool = False,
    ) -> None:
        """Init params."""
        super().__init__()
        self._llm = llm or Settings.llm
        self._decompose_query_prompt = (
            decompose_query_prompt or DEFAULT_DECOMPOSE_QUERY_TRANSFORM_PROMPT
        )
        self.verbose = verbose

    ...
    
    def _run(self, query_bundle: QueryBundle, metadata: Dict) -> QueryBundle:
        """Run query transform."""
        index_summary = cast(str, metadata.get("index_summary", "None"))
        query_str = query_bundle.query_str
        new_query_str = self._llm.predict(
            self._decompose_query_prompt,
            query_str=query_str,
            context_str=index_summary,
        )

        if self.verbose:
            print_text(f"> Current query: {query_str}\n", color="yellow")
            print_text(f"> New query: {new_query_str}\n", color="pink")

        return QueryBundle(
            query_str=new_query_str,
            custom_embedding_strs=[new_query_str],
        )

Ensure that:

1. The verbose parameter is set to True when initializing the DecomposeQueryTransform object.
2. The print_text function is correctly defined and imported.

Example initialization:

query_transform = DecomposeQueryTransform(verbose=True)

If the verbose parameter is correctly set and the print_text function is working, the sub-queries should be printed as expected ^[1].

Additionally, verify that the print_text function is defined and imported correctly. It is defined in the llama_index.legacy.utils module:

def print_text(text: str, color: Optional[str] = None, end: str = "") -> None:
    text_to_print = _get_colored_text(text, color) if color is not None else text
    print(text_to_print, end=end)

Ensure that this function is available in your environment ^[2].

To continue talking to Dosu, mention @dosu.