Dedup ID on Pinecone DB

[ranggaaldosas]

I’m currently using the Pinecone connector in Airbyte with Incremental | Append + Deduped mode, and I’ve set cve_id as the primary key. However, I’m still seeing duplicate vectors in my index for the same CVE. Each update is creatin/upserting a new vector instead of update the existing one.

When I inspect the index, I can see that the connector is using the Airbyte-generated record IDs as the Pinecone vector IDs, for example:

'id': 'cve_records#632e5bda-0036-4c03-8934-0bc6d19da444',
'metadata': { '_ab_record_id': 'cve_records_cve_records_CVE-2024-38211', … }

'id': 'cve_records#01784779-08d9-4212-bd3e-64f3f712ccf5',
'metadata': { '_ab_record_id': 'cve_records_cve_records_CVE-2024-38211', … }

Instead of upserting on cve_id, it’s assigning those random UUIDs (632e5bda-…, 01784779-…) as the vector IDs. Can you please advise:

1. Why the Append + Deduped mode isn’t deduplicating by cve_id?
2. How I can configure the connector so that it uses my cve_id field directly as the Pinecone vector ID?

Thanks for your help!