Efficient Filtering Strategy in Zilliz When User Selects Large Dynamic File Sets

[kamelabiad]

We are facing a design challenge related to filtering vector search results in Zilliz at scale.

The issue is not how to send data to Zilliz, but rather how to efficiently restrict search results to a large, dynamic subset of files selected by the user at query time.

Scenario
Total files indexed in Zilliz: ~1,000,000
Each file has one or more embeddings stored in Zilliz

A user:
Has access to all 1M files
Can dynamically select up to 200,000 files per search
The selection can change frequently between searches
Current Limitation
We cannot use a filter like:

fileId IN (1,2,3,...)

because:

Filter expressions have size limits
Performance degrades significantly with very large IN lists
Core Question
What is the recommended or best-practice approach in Zilliz to perform vector search only within a large, dynamic user-selected subset of entities (e.g. 200k out of 1M), without passing large ID lists or frequently updating vector metadata?

[xiaofan-luan]

usually we recommend people to directly to store their meta into milvus, not other system so filter can be much easier.

if there is a reason you must do so, try filtering template, might be a little bit faster on expression parse.

[kamelabiad]

The reason is that users have access to an unlimited number of files, and each time they can select any subset of files from that set.

[yhmo]

You can use filter template to pass the 200,000 ids
https://milvus.io/docs/filtering-templating.md#Filter-Templating

results = client.search(collection_name=COLLECTION_NAME,
                            data=[vector_to_search],
                            limit=10,
                            filter='fileId in {values}',
                            filter_params={"values": [1, 2, 3, ......]},
                            )

The "filter_params" passes the values by list instead of string-format, is much efficient than string expression.

For one million integer ids, the size is 8bytes * 1M = 8MB, is much smaller than the grpc size limit.

[xiaofan-luan]

As long as the request is less than 16MB, it should work. However, having too many filters may slow things down.

@kamelabiad
I’d like to better understand your use case. Are you building a RAG system where users can dynamically select a subset of files to act as the knowledge base? Is that a correct understanding?

[kamelabiad]

@xiaofan-luan Yes thats exactly the case , users can select any files/folders randomly from their own files list

[xiaofan-luan]

what could be typical ID numbers in this filter?

if it's always under 1m, I think you can simply run with expression template

[kamelabiad]

Yes file Id is a number , but will this affect the performance ? will it take alot of time if we implement this filter expression on this huge number of files

[yhmo]

Yes file Id is a number , but will this affect the performance ? will it take alot of time if we implement this filter expression on this huge number of files

Filter template reduce the time cost to parse long-length string expression, so it is faster than pure-string expression.
I just compare the time cost between filter template and string expression in a small dataset(only 1 rows):

values = [k for k in range(1000000)]
# filter template
t1 = time.time()
results = client.search(collection_name=COLLECTION_NAME,
                        data=[vector_to_search],
                        limit=10,
                        filter='flag in {values}',
                        filter_params={"values": values},
                        consistency_level="Bounded",
                        )
t2 = time.time()
print("filter template time cost: ", (t2 - t1) * 1000, "ms")

t1 = time.time()
results = client.search(collection_name=COLLECTION_NAME,
                        data=[vector_to_search],
                        limit=10,
                        filter=f'flag in {values}',
                        consistency_level="Bounded",
                        )
t2 = time.time()
print("string expression time cost: ", (t2 - t1) * 1000, "ms")

The "values" is a 1M ids list. The average time cost of filter template is around 800ms, average time cost of string expression is around 1200ms.

So, huge number of files will add at least 800ms to search latency.

[kamelabiad]

Hi Again @yhmo ,we are trying this code but it looks like it is not filtering and giving this error message ""[code, 1100]"
"[message, failed to create query plan: the value of expression template variable name {fileIds} is not found: invalid parameter]""

{

  "collectionName": "file_summary_cosine",
  "outputFields": [
    "id",
    "FileId",
    "text"
  ],

  "limit": 30,
  "search": [
    {
      "annsField": "text_dense",
      "data": [
        [
          -0.011806186,
          ...,
          -0.0038755997
        ]
      ],

      "limit": 60,

      "metricType": "COSINE",

      "params": {

        "nprobe": 16

      },

      "filter": "FileId IN {fileIds}",

      "filter_params": {

        "fileIds": [

          13889

        ],

        "sectionIndices": null

      }

    },

    {

      "annsField": "text_sparse",

      "data": [

        "what color is the sky"

      ],

      "limit": 60,

      "params": {},

      "filter": "FileId IN {fileIds}",

      "filter_params": {

        "fileIds": [

          13889

        ],

        "sectionIndices": null

      }

    }

  ],

  "rerank": {

    "strategy": "rrf",

    "params": {

      "k": 60

    }

  }

}

[yhmo]

"filter template" is available from milvus v2.5.0, maybe your milvus version is lower than 2.5.0

[kamelabiad]

We are using zillis cloud version

[yhmo]

Change the "filter_params" to be "exprParams":

"filter_params": {
        "fileIds": [
          13889
        ],
        "sectionIndices": null
      }

"exprParams": {
        "fileIds": [13889]
}

Sorry, we missed this parameter in our document, we will fix the doc later.