Error inserting DoclingLoader documents on Weaviate

[alex-ahumada]

DoclingLoader uses a non valid GraphQL name in one of its Document metadata keys ($ref) that causes an error while saving the documents into Weaviate.

{'message': 'Failed to send all objects in a batch of 17', 'error': 'WeaviateInsertManyAllFailedError("Every object failed during insertion. Here is the set of all errors: \'$ref\' is not a valid nested property name of \'dl_meta.doc_items.parent\'. NestedProperty names in Weaviate are restricted to valid GraphQL names, which must be “/[_A-Za-z][_0-9A-Za-z]*/”")'}
{'message': 'Failed to send 17 objects in a batch of 17. Please inspect client.batch.failed_objects or collection.batch.failed_objects for the failed objects.'}
2025-Jul-04 10:50 AM - langchain_weaviate.vectorstores - ERROR - Failed to add object: None
Reason: WeaviateInsertManyAllFailedError("Every object failed during insertion. Here is the set of all errors: '$ref' is not a valid nested property name of 'dl_meta.doc_items.parent'. NestedProperty names in Weaviate are restricted to valid GraphQL names, which must be “/[_A-Za-z][_0-9A-Za-z]*/”")

After deleting the offending metadata or renaming $ref key to ref, documents are inserted successfully.

[alex-ahumada]

I've solved it temporarily by using a custom MetaExtractor.

class CustomMetaExtractor(BaseMetaExtractor):
    """MetaExtractor."""
    
    def sanitize_json_keys(self, obj):
        """
        Recursively remove any '$' characters from all dict keys in a JSON-like structure.
        """
        if isinstance(obj, dict):
            new_dict = {}
            for k, v in obj.items():
                new_key = k.replace("$", "")
                new_dict[new_key] = self.sanitize_json_keys(v)
            return new_dict
        elif isinstance(obj, list):
            return [self.sanitize_json_keys(item) for item in obj]
        else:
            return obj

    def extract_chunk_meta(self, file_path: str, chunk: BaseChunk) -> dict[str, Any]:
        """Extract chunk meta."""
        return {
            "source": file_path,
            "dl_meta": self.sanitize_json_keys(chunk.meta.export_json_dict()),
        }

    def extract_dl_doc_meta(
        self, file_path: str, dl_doc: DoclingDocument
    ) -> dict[str, Any]:
        """Extract Docling document meta."""
        return {"source": file_path}

loader = DoclingLoader(
    file_path=FILE_PATH,
    export_type=EXPORT_TYPE,
    chunker=HybridChunker(
        tokenizer=HuggingFaceTokenizer.from_pretrained(
            model_name=EMBED_MODEL_ID,
        ),
    ),
    meta_extractor=CustomMetaExtractor(),
)