Skip to content

Add support for ingesting Confluence.  #57

@homanp

Description

@homanp

Describe the issue

A user should be able to ingest documents from confluence by passing the following payload to the /ingest API endpoint:

{
  "index_name": "string",
  "vector_database": {
    "type": "qdrant",
    "config": {}
  },
  "encoder": {
      "dimensions": 384,
      "model_name": "embed-multilingual-light-v3.0",
      "provider": "cohere"
    },
  "confluence": {
    "api_token": <api_token>,
    "user_email": < user_email >,
    "url":  <url>,
  },
  "webhook_url": "string"
}

Ingestion

After load process has finished we need to loop over the downloaded documents and use the embedding service to chunk, split, generate and save embeddings to the embedding db. The Embedding Service and walkthroughs should be sufficient for this.

Relevant files

Route ingestion on request: https://github.com/superagent-ai/super-rag/blob/main/api/ingest.py#L23
Create a new handler: https://github.com/superagent-ai/super-rag/blob/main/service/ingest.py

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions