Lost of Metadata during adding documents to postgresSQL vectorstore

[charlckh]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

### qa_tobe_added is a dictionary of a vectorstore id mapping to list of documents to be added to that vectorstore. Vectorstores are google cloud PostgreSql vectorstores
    
        for i in qa_tobeadded:
            if qa_tobeadded[i] == []:
                break
            print(f"Adding documents for {i} vectorstore")
            print(i)
            vectorstore = get_PostgreSQLvectorstore(i)
            docs_to_add = qa_tobeadded[i]
            print(docs_to_add)
            times =  len(docs_to_add) / 50
            times = math.ceil(times)

            for j in range(times):
                init_index = 50*j
                if init_index+50 <= len(docs_to_add):
                    splitted_qa_docs = docs_to_add[init_index: init_index+50]
                    vectorstore.add_documents(splitted_qa_docs)
                    print("Added 50 documents")
                else:

                    splitted_qa_docs = docs_to_add[init_index:]
                    print(splitted_qa_docs)

                    ids_to_be_added = qa_ids[init_index:]
                    vectorstore.add_documents(splitted_qa_docs)

                    print("Added remaining documents")

Description

Whenever there is two identical documents in each key, only one of them can be added with correct metadata to the vectorstore. The remaining one will lost the metadata which should be added into the columns of vectorstore.
Example:
Original qa_tobe_added :
{'2': [Document(metadata={'source': 'QnA', 'question': '如何將處方從正藥改為副廠藥?', 'doc_id': '1'}, page_content='Question:如何將處方 從正藥改為副廠藥? Answer:你可以請醫生在處方上註明"Generic drug is allowed" 並在側面簽字或 處方中只能列出藥品的化學名稱，避免指定任何正廠或製造商；這也允許使用副廠藥。')], '1': [Document(metadata={'source': 'QnA', 'question': '如何將處方從正藥改為副廠藥?', 'doc_id': '1'}, 商；這也允許使用副廠藥。')]

After adding the document to one of the vectorstore:
{'2': [Document(metadata={'question': '如何將處方從正藥改為副廠藥?'}, page_content='Question:如何將處方從正藥改為副廠藥? Answer:你可以請醫生在處方上註明"Generic drug is allowed" 並在側面簽字或 處方中只能列出藥品的化學名稱，避免指定任何正廠或製造商；這也允許使用副廠藥。')], '1': [Document(metadata={'question': '如何將處方從正藥改為副廠藥?'}, page_content='Question:如何將處方從正藥改為副廠藥? Answer:你可以 請醫生在處方上註明"Generic drug is allowed" 並在側面簽字或 處方中只能列出藥品的化學名稱，避免指定任何正廠或製造商；這也允許使用副廠藥。')]}

As we observed, metadata source and doc_id are lost

System Info

langchain_core: 0.3.51
langchain: 0.3.23
langchain_community: 0.3.21
langsmith: 0.2.3
langchain_chroma: 0.2.2
langchain_docling: 0.2.0
langchain_google_alloydb_pg: 0.10.0
langchain_google_cloud_sql_pg: 0.13.0
langchain_google_community: 2.0.7
langchain_google_vertexai: 2.0.20
langchain_text_splitters: 0.3.8
langchain_unstructured: 0.1.6
langgraph_sdk: 0.1.45