BasicRAGPipeline

Tried all the data ingestion techniques.

Learned to ingest text file using textloader of langchain.

Ingested data from a webpage after scraping it through beautifulsoup package in python.

Also, ingested data from pdfs using PyPDFloader package

Split the data into chunks to feed to the model.

Used OpenAI embeddings for vector embeddings.

Used chroma DB as well as FAISS (both) to store those vectors and hit them to retrieve the results.