Staging To Main (#190)

* added types for API response

* type fixes

* Wiki issue fixed

* connect URL

* Chatbot cross icon visibility

* Update gemini_llm.py

* Update gemini_llm.py

* Update the library google-cloud-aiplatform

* Update logging level from Info to DEBUG

* sources req

* Update gemini_llm.py

* Update gemini_llm.py

* Update Dockerfile

* Update Dockerfile

* Update Dockerfile

* review comments solution

* Update Dockerfile

* Update Dockerfile

* query changes

* dockerfile changes

* Update gemini_llm.py

* dockerfile changes

* Update gemini_llm.py

* query changes for entity

* format and lint fixes

* fixed doc no issue

* removed settings icon

* gemini model changes using langserve

* LLM Model Changes

* llm model changes

* response changes

* update vector index

* update logging to track embeddings

* connection modal changes

* table-query-check

* MOve create source node try catch in score.py

* create seprate function for each source URL

* Gemini security remove and added sleep

* Update requirements.txt

* URL scan Code refactoring

* query changes

* new connect modal

* Update Extract function

* Update requirements.txt

* Extract API code improvement

* folder name correction for gcs bucket

* Remove unused function and move make_relationship function to other class

* Added sentence-transformer embeddings and update read file as temp from local

* Gemini security remove and added sleep

* Update requirements.txt

* Revert "Update requirements.txt"

This reverts commit 7237be5.

* Merging issue fixed

* combining 4 chunks

* driver-changes

* updated list name

* made chunks to combine configurable

* added multiple models and embeddings for chatbot

* fixed configurable source issue

* added model param for  QNA API

* lint fixes

* Added Common function

* modified prompt

* fixed chat response  issue

* modal changes

* merged Update modal

* llm dropdown changes

* session changes

* prefilling the input field fix

* modified prompt

* Modified prompt

* added env variable for llms

* Modified prompt and added sources to response

* Modified output response structure

* path-change

* added env values to docker file

* format fixes

* graph fix

* Modified source extracting function

* added source types

* added sources links in the chat bot message

* conditional source check

* File uploaded into chunks

* function name change

* format fixes

* added chunking frontend code

* added new status

* Modified prompt

* added new status after upload completion

* Modified prompt

* Issue fixed relation to embeding model and large file upload

* added progress bar

* added progress staus in table

* legends check

* removed binary filesstate

* backend processing source fix

* alert fix

* Modified commit

* openai error resolve

* added file name for all sources

* Issues Fixed

* Gemini versions and library realted changes updated

* Update Model details

* Updated prompt

* Import graphDbClass on score.py

* posix path error fix in pathlib

* changed chat bot icon

* added Gemini 1.5Pro

* color changes

* format fixes

* backend url change

---------

Co-authored-by: kartikpersistent <[email protected]>
Co-authored-by: Pravesh Kumar <[email protected]>
Co-authored-by: Prakriti Solankey <[email protected]>
Co-authored-by: karanchellani <[email protected]>
Co-authored-by: vasanthasaikalluri <[email protected]>

Author: 6 people

.gitignore

@@ -162,4 +162,6 @@ cython_debug/
 temp.pdf
 google-cloud-sdk
 google-cloud-cli-469.0.0-linux-x86_64.tar.gz
-/data/llm-experiments-387609-c73d512ca3b1.json
+/data/llm-experiments-387609-c73d512ca3b1.json
+/backend/src/merged_files
+/backend/src/chunks

README.md

@@ -74,7 +74,7 @@ KNN_MIN_SCORE = ""\
    	 userName: Username to use for graph creation ( if None will use username from config file )
    	 password: Password to use for graph creation ( if None will use password from config file )
    	 file: File object containing the PDF file path to be used
-   	 model: Type of model to use ('OpenAI GPT 3.5' or 'OpenAI GPT 4')
+   	 model: Type of model to use ('Gemini Pro' or 'Diffbot')
    
      Returns: 
    	 Json response to API with fileName, nodeCount, relationshipCount, processingTime, 
@@ -114,7 +114,7 @@ KNN_MIN_SCORE = ""\
 https://github.com/neo4j-labs/llm-graph-builder/assets/121786590/b725a503-6ade-46d2-9e70-61d57443c311
 
 ## Links
- The Public [ Google cloud Run URL](https://staging-frontend-dcavk67s4a-uc.a.run.app/).
+ The Public [ Google cloud Run URL](https://devfrontend-dcavk67s4a-uc.a.run.app).
  [Workspace URL](https://workspace-preview.neo4j.io/workspace)
 
 

backend/Dockerfile

@@ -9,4 +9,6 @@ RUN apt-get update \
     && apt install -y tesseract-ocr \
     && export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH \
     && pip install --no-cache-dir --upgrade -r /code/requirements.txt
-CMD ["uvicorn", "score:app", "--host", "0.0.0.0", "--port", "8000"]
+    
+CMD ["uvicorn", "score:app", "--host", "0.0.0.0", "--port", "8000"]
+

backend/example.env

@@ -12,4 +12,5 @@ KNN_MIN_SCORE = ""
 LANGCHAIN_API_KEY = ""
 LANGCHAIN_PROJECT = ""
 LANGCHAIN_TRACING_V2 = ""
-LANGCHAIN_ENDPOINT = ""
+LANGCHAIN_ENDPOINT = ""
+NUMBER_OF_CHUNKS_TO_COMBINE = ""

backend/requirements.txt

@@ -28,8 +28,8 @@ docstring_parser==0.16
 effdet==0.4.1
 emoji==2.10.1
 exceptiongroup==1.2.0
-fastapi==0.110.0
-fastapi-health==0.4.0
+fastapi
+fastapi-health
 filelock==3.13.1
 filetype==1.2.0
 flatbuffers==23.5.26
@@ -38,7 +38,7 @@ frozenlist==1.4.1
 fsspec==2024.2.0
 google-api-core==2.18.0
 google-auth==2.29.0
-google-cloud-aiplatform==1.45.0
+google-cloud-aiplatform
 google-cloud-bigquery==3.19.0
 google-cloud-core==2.4.1
 google-cloud-resource-manager==1.12.3
@@ -78,6 +78,8 @@ langchain-text-splitters==0.0.1
 langdetect==1.0.9
 langsmith==0.1.31
 layoutparser==0.3.4
+langserve
+langchain-cli
 lxml==5.1.0
 MarkupSafe==2.1.5
 marshmallow==3.20.2
@@ -156,9 +158,10 @@ unstructured-client
 unstructured-inference
 unstructured.pytesseract
 urllib3
-uvicorn==0.29.0
+uvicorn
 wikipedia==1.4.0
 wrapt==1.16.0
 yarl==1.9.4
 youtube-transcript-api==0.6.2
 zipp==3.17.0
+sentence-transformers