[BUG] on_disk_payload always return None even if it is explicitly provided value from argument

[bibekyess]

Hi!
I noticed that when I am using qdrant locally then on_disk_payload parameter cannot be set to True, which makes the RAM usage very high while indexing large number of documents.
For my usecase, I need to use local qdrant instance without docker, so I am wondering if this behaviour is expected or a bug.

Replication code:

from qdrant_client import QdrantClient, models

client_path=".../local-path"
qdrant_client = QdrantClient(path=client_path,)
collection_name="test"
qdrant_client.create_collection(
            collection_name=collection_name,
            vectors_config= models.VectorParams(
                    size=1024,
                    distance=models.Distance.COSINE,
                    on_disk=True,
                ),
            optimizers_config=models.OptimizersConfigDiff(memmap_threshold=10000),
            hnsw_config=models.HnswConfigDiff(on_disk=True),
            on_disk_payload=True # This is not working
)

print(vars(qdrant_client.get_collection(collection_name).config))

Output

{'params': CollectionParams(vectors=VectorParams(size=1024, distance=<Distance.COSINE: 'Cosine'>, hnsw_config=None, quantization_config=None, on_disk=True, datatype=None, multivector_config=None), shard_number=None, sharding_method=None, replication_factor=None, write_consistency_factor=None, read_fan_out_factor=None, on_disk_payload=None, sparse_vectors=None), 'hnsw_config': HnswConfig(m=16, ef_construct=100, full_scan_threshold=10000, max_indexing_threads=0, on_disk=None, payload_m=None), 'optimizer_config': OptimizersConfig(deleted_threshold=0.2, vacuum_min_vector_number=1000, default_segment_number=0, max_segment_size=None, memmap_threshold=None, indexing_threshold=20000, flush_interval_sec=5, max_optimization_threads=1), 'wal_config': WalConfig(wal_capacity_mb=32, wal_segments_ahead=0), 'quantization_config': None}

Output with docker image

{'params': CollectionParams(vectors=VectorParams(size=1024, distance=<Distance.COSINE: 'Cosine'>, hnsw_config=None, quantization_config=None, on_disk=True, datatype=None, multivector_config=None), shard_number=1, sharding_method=None, replication_factor=1, write_consistency_factor=1, read_fan_out_factor=None, on_disk_payload=True, sparse_vectors=None), 'hnsw_config': HnswConfig(m=16, ef_construct=100, full_scan_threshold=10000, max_indexing_threads=0, on_disk=True, payload_m=None), 'optimizer_config': OptimizersConfig(deleted_threshold=0.2, vacuum_min_vector_number=1000, default_segment_number=0, max_segment_size=None, memmap_threshold=10000, indexing_threshold=20000, flush_interval_sec=5, max_optimization_threads=None), 'wal_config': WalConfig(wal_capacity_mb=32, wal_segments_ahead=0), 'quantization_config': None}

Specs:
Name: qdrant-client
Version: 1.10.1

[joein]

Hi @bibekyess

You're using Qdrant in local mode, which is not supposed to be used for indexing a large amount of documents
It has been built to be used in small experiments before switching to a server version (e.g. in docker container)

Some of the functionality is not supported there, however, the interface is the same as in the server version, so you could switch to it just by replacing path= to url=

[bibekyess]

Hi @joein Thank you for your response!
Yes, I also considered running via docker container but the issue still persists.
Can you find this page with the replication code attached.
run-llama/llama_index#15013 (comment)
Thanks!