Index on nested payload field?

[mymindwentblvnk]

Hi, do you know if it is possible to create an index on a nested payload field?

As written here (https://qdrant.tech/documentation/concepts/indexing/#payload-index), one can easily create an index on a top level field. Our payload looks like this:

{
    'id': 1,
    'vector': ...,
    'payload': {
        'top_level_attribute': [
            { 'nested_attribute': ['s1', 's2', 's3', 's4', ...], 'other_nested_attribute': 'some_string' },
            { 'nested_attribute': ['s3', 's4', ...], 'other_nested_attribute': 'some_other_string' },
        ]
    }
}

I want to filter on a certain string in nested_attribute. How could I create an index on it?

[mymindwentblvnk]

So the following index definition seems to work:

client.create_payload_index('collection_1', field_name='top_level_attribute[].nested_attribute', field_schema='keyword')

When checking the collection information, it confirms to have created an index for all points in the collection.

collection_info = client.get_collection('collection_1')
assert collection_info.payload_schema['top_level_attribut[].nested_attribute'].points == number_of_points_in_collection  # This is True

But I cannot measure any performance improvement.
May be field_schema='keyword' is not the correct type, since it is a list of keywords?

[gururaser]

Hi, @mymindwentblvnk have you been able to verify practically that this method works? I feel like If I don't see "filter" logo in Qdrant Cloud UI (you can see it in platform_brand_name), it doesn't have index even though I see on collection information:

self.qm.create_payload_index("platform_brand_name", models.TextIndexParams(type=models.TextIndexType.TEXT, tokenizer= models.TokenizerType.WORD, min_token_len=2, max_token_len=20, lowercase=True, on_disk=True))
self.qm.create_payload_index("scores.rating", models.FloatIndexParams(type="float", is_principal=True))
self.qm.create_payload_index("scores.merchant", models.FloatIndexParams(type="float", is_principal=True))

def create_payload_index(self, field_name: str, field_schema: Any) -> bool:
        try:
            target_collection = self.get_target_collection()
            collection_info = self.client.get_collection(target_collection)
            existing_indices = collection_info.payload_schema

            if field_name not in existing_indices:
                self.client.create_payload_index(
                    collection_name=target_collection,
                    field_name=field_name,
                    field_schema=field_schema
                )
                logger.info(f"Index created for field '{field_name}' in collection '{target_collection}'.")
            else:
                logger.info(f"Index already exists for field '{field_name}' in collection '{target_collection}'. Skipping creation.")
            return True
        except Exception as e:
            logger.error(f"Error occurred while checking/creating index for field '{field_name}': {e}")
            return False

[espdev]

It seems it works now (version='1.14.1'). At least I notice a significant increase in filtering performance on indexed nested payload fields.

[espdev]

Still, there seems to be a problem with the index for nested filters.
When we use a normal payload filter, the index works very fast, but when we apply a nested object filter, the index doesn't work or works slowly, selectively and oddly.

For example, we have a full text search index for the field records[].address.

Apply a normal filter:
POST collections/my_collection/points/count

{
  "filter": {
    "must": [
      {
        "key": "records[].address",
        "match": {
          "text": "peters st."
        }
      }
    ]
  }
}

Result:

{
  "result": {
    "count": 66
  },
  "status": "ok",
  "time": 0.001036682
}

Apply a nested object filter:

{
  "filter": {
    "must": [
      {
        "nested": {
          "key": "records",
          "filter": {
            "must": [
              {
                "key": "address",
                "match": {
                  "text": "peters st."
                }
              }
            ]
          }
        }
      }
    ]
  }
}

Result

{
  "result": {
    "count": 66
  },
  "status": "ok",
  "time": 0.416035051
}

A normal filter on this example is about 400 times faster.
The difference in execution time between a regular filter and a nested filter can reach hundreds and thousands of times for our data.

[generall]

hey @espdev, can you share a collection snapshot, where we can reproduce this?

[espdev]

Hi @generall,

Unfortunately, I can't share the data as it is private data that cannot be disclosed.

However the format of our data is like this:

{
  "payload": {
    "records": [
      {"address": "1560 N. Peters St.", "city": "New Orleans", "state": "LA", "zip": "60120", ...},
      {"address": "6070 Regency St.", "city": "Cary", "state": "NC", "zip": "35627", ...},
      ...
    ]
  }
}

We have about 75000 points in the collection, and each point contains a payload with 1-700 records. Each record contains ~40 fields with types: string, integer, datetime, geo. Indexes have been created for almost all fields.

We need to filter on the fields in the records when we do a vector search.

[generall]

can you generate a small amount of fake data with the exact format, indexes and filters you are using?

[espdev]

@generall
Yeah, I think I can generate fake data. I'll try to do it soon.
Thank you for your attention!