Skip to content

Commit 6b81540

Browse files
anmarhindiLennartSchmidtKernJWittmeyerlumburovskalina
authored
FastAPI Integration (#196)
* adds createOutlierSlice * adds recordsByStaticSlice * comment fix * record export * getRecordExportFromData * adds staticDataSlicesCurrentCount * removes unused import * remove unused imports * adds availableLinks * adds createDataSlice * adds lastRecordExportCredentials * fix gates integration data for updating projects * fix status ref * get_gates_integration_data patch * adds prepareRecordExport + changes to lastRecordExportCredentials * adds requestHuddleData * Create label mutation * Record label association by ids * Delete record by recordId * adds getTokenizedRecord * adds Request json call inside try body * Search records by similarity search * Merge * changes huddle id source * Get labeling function on 10 records * changes * Zero shot text query * Merge * Zero shot on 10 records * Export lookup list query * adds recordByRecordId * patch url structure * Model callbacks query * Merge * Can create local org query * accessLink * Personal tokens query * isRatsTokenizationStillRunning * lastProjectExportCredentials * added linkLocked + simple pydantic model to validate body * clean line * Project size query * Create new labels mutation * add pydantic models * remove import * calculateUserAttributeSampleRecords * adds notifications * uploadCredentialsAndId mutation * RecordIdeBody * deleteDataSliceById * editRecords * toggleInformationSource * adds updateDataSlice * uploadTaskById * edit linkLocked, remove unused example model * deleteInformationSource * setAllInformationSourceSelected, zeroShotProject, createPayload, initiateWeakSupervisionByProjectId * createHeuristic * createZeroShotInformationSource * updateInformationSource * adds generateAccessLink + fixes * remove swp file * runHeuristicThenTriggerWeakSupervision * cancelZeroShotRun * adds removeAccessLink * adds lockAccessLink * remove unused * addClassificationLabelsToRecord * addExtractionLabelToRecord * setGoldStarAnnotationForTask and removeGoldStarAnnotationForTask * createOrganization and addUserToOrganization * changeOrganization * updateConfig * Enforces project access checks on endpoints that use project_id, prevents bypass * createPersonalAccessToken and deletePersonalAccessToken * createUserAttribute * remove print statement * adds createKnowledgeBase * remove duplicate path from get_export_lookup_list * removed and tested duplicate path for get_terms_by_lookup_list_id * tested/removed/updated duplicate path for get_lookup_lists_by_lookup_list_id * tested/removed/updated duplicate path for get_lookup_lists_by_lookup_list_id * tested/removed/updated duplicate path for get-lookup-lists-by-project-id * updateAttribute * adds updateKnowledgeBase * updateProjectNameAndDescription * adds deleteKnowledgeBase * updateProjectForGates * Integrate project access checks into router dependencies for attribute route * Integrate project access checks into router dependencies for data_browser and data_slices routes * Integrate project access checks into router dependencies for heuristic, organization and zero_shot routes * Integrate project access checks into router dependencies for labeling, lookup_lists and project setting routes * Integrate project access checks into router dependencies for project, record_ide, record, weak_supervision and zero_shot routes * deleteFromTaskQueue and deleteEmbedding * createEmbedding * adds addTermToKnowledgeBase * adds deleteTerm * adds blacklistTerm * adds pasteKnowledgeTerms * adds updateTerm * adds updateEmbeddingPayload * Update project access check for data_slices * Update project access check for embedding * typo * Update project access check for lookup_lists * Update project access check for project_setting * Update project access check for project * remove unused import * adds updateLabelingTask * update notifications * updates project access for routes * adds deleteLabelingTask * adds createLabelingTask * adds deleteLabel * changes to labeling * adds updateLabelColor * adds updateLabelHotkey * adds handleLabelRenameWarnings * adds updateLabelName * adds createComment * adds deleteComment * adds updateComment * patch all-comments * minor clean * adds deleteProject * adds createProject * adds updateProjectTokenizer, patch getTokenizedRecord * renames get-data-slices * fixes weak-supervision-run * adds updateProjectStatus * adds createSampleProject * adds calculateUserAttributeAllRecords * adds createTaskAndLabels * adds modelProviderDeleteModel * adds modelProviderDownloadModel * adds deleteUserAttribute * adds prepareProjectExport * Fixes heuristic request with most recent payload * Fixes embedding filter attributes wrapping & unique values * merge with dev * Adds gql wrapper for record ide request * Rename endpoint to better represent the task * allows optional values for lookuplist update * Submodule change * Fixes label deletion * Removes onQdrant from whitelist * Fixes data slice update * Fixes slice collection * Removes unused whitelists * Fixes missing dependencies * adds recordLabelAssociations * remove print * Labeling tasks requests into a separate file * PR comments * PR comments * fixes similarity without filter issues * Fixes data slice request structure * Unasync prep project endpoint * Small fix for updating data slices * patch embeddings-by-project * resolve * clean * removes async * upgrades get_all_comments * remove unused imports * upgrades get_record_comments, additions to get_embeddings * remove unused imports * updates zero-shot-10-records * updates zero-shot-text * removes unused imports * updates create-labels * removes duplicate import * upgrades create_label * remove unused import * updates delete_data_slice_by_id * upgrades update_data_slice * upgrades available-links * upgrades huddle-data * edits available-links * upgrades delete_record_label_association_by_ids * removes unused import * upgrades get_tokenized_record * upgrades create_data_slice * remove unused import * upgrades get_search_records_by_similarity * huddle edits * upgrade search-records-extended * upgrades records-by-static-slice * Fixed labeling function on 10 records for extraction * remove unused import * edit * Hotfix session deletion * Fixed issue for only different results * patch record-label-associations * update model ref * SUbmodules merged * Revert "SUbmodules merged" This reverts commit d4f194d. * Submodules merge * Merge * Submodules merge --------- Co-authored-by: LennartSchmidtKern <[email protected]> Co-authored-by: JWittmeyer <[email protected]> Co-authored-by: Lina <[email protected]> Co-authored-by: JWittmeyer <[email protected]>
1 parent de42d74 commit 6b81540

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

55 files changed

+4590
-150
lines changed

‎app.py‎

Lines changed: 88 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
import logging
2+
3+
from fastapi import FastAPI
24
from api.healthcheck import Healthcheck
35
from api.misc import IsDemoRest, IsManagedRest
46
import graphene
@@ -15,21 +17,104 @@
1517
CognitionPrepareProject,
1618
CognitionParseMarkdownFile,
1719
)
20+
from fast_api.routes.organization import router as org_router
21+
from fast_api.routes.project import router as project_router
22+
from fast_api.routes.project_setting import router as project_setting_router
23+
from fast_api.routes.misc import router as misc_router
24+
from fast_api.routes.comment import router as comment_router
25+
from fast_api.routes.zero_shot import router as zero_shot_router
26+
from fast_api.routes.attribute import router as attribute_router
27+
from fast_api.routes.embedding import router as embedding_router
28+
from fast_api.routes.notification import router as notification_router
29+
from fast_api.routes.data_slices import router as data_slice_router
30+
from fast_api.routes.lookup_lists import router as lookup_lists_router
31+
from fast_api.routes.heuristic import router as heuristic_router
32+
from fast_api.routes.data_browser import router as data_browser_router
33+
from fast_api.routes.labeling import router as labeling_router
34+
from fast_api.routes.record_ide import router as record_ide_router
35+
from fast_api.routes.record import router as record_router
36+
from fast_api.routes.weak_supervision import router as weak_supervision_router
37+
from fast_api.routes.labeling_tasks import router as labeling_tasks_router
1838
from middleware.database_session import DatabaseSessionHandler
1939
from starlette.applications import Starlette
2040
from starlette.graphql import GraphQLApp
2141
from starlette.middleware import Middleware
22-
from starlette.routing import Route
42+
from starlette.routing import Route, Mount
2343

2444
from graphql_api import schema
2545
from controller.task_queue.task_queue import init_task_queues
2646
from controller.project.manager import check_in_deletion_projects
47+
from route_prefix import (
48+
PREFIX_ORGANIZATION,
49+
PREFIX_PROJECT,
50+
PREFIX_PROJECT_SETTING,
51+
PREFIX_MISC,
52+
PREFIX_COMMENT,
53+
PREFIX_ZERO_SHOT,
54+
PREFIX_ATTRIBUTE,
55+
PREFIX_EMBEDDING,
56+
PREFIX_NOTIFICATION,
57+
PREFIX_DATA_SLICE,
58+
PREFIX_LOOKUP_LISTS,
59+
PREFIX_HEURISTIC,
60+
PREFIX_DATA_BROWSER,
61+
PREFIX_LABELING,
62+
PREFIX_RECORD_IDE,
63+
PREFIX_RECORD,
64+
PREFIX_WEAK_SUPERVISION,
65+
PREFIX_LABELING_TASKS,
66+
)
2767
from util import security, clean_up
2868

29-
3069
logging.basicConfig(level=logging.DEBUG)
3170
logger = logging.getLogger(__name__)
3271

72+
fastapi_app = FastAPI()
73+
74+
fastapi_app.include_router(
75+
org_router, prefix=PREFIX_ORGANIZATION, tags=["organization"]
76+
)
77+
fastapi_app.include_router(project_router, prefix=PREFIX_PROJECT, tags=["project"])
78+
fastapi_app.include_router(
79+
project_setting_router, prefix=PREFIX_PROJECT_SETTING, tags=["project-setting"]
80+
)
81+
fastapi_app.include_router(misc_router, prefix=PREFIX_MISC, tags=["misc"])
82+
fastapi_app.include_router(comment_router, prefix=PREFIX_COMMENT, tags=["comment"])
83+
fastapi_app.include_router(
84+
zero_shot_router, prefix=PREFIX_ZERO_SHOT, tags=["zero-shot"]
85+
)
86+
fastapi_app.include_router(
87+
attribute_router, prefix=PREFIX_ATTRIBUTE, tags=["attribute"]
88+
)
89+
fastapi_app.include_router(
90+
embedding_router, prefix=PREFIX_EMBEDDING, tags=["embedding"]
91+
)
92+
fastapi_app.include_router(
93+
notification_router, prefix=PREFIX_NOTIFICATION, tags=["notification"]
94+
)
95+
fastapi_app.include_router(
96+
data_slice_router, prefix=PREFIX_DATA_SLICE, tags=["data-slice"]
97+
)
98+
fastapi_app.include_router(
99+
lookup_lists_router, prefix=PREFIX_LOOKUP_LISTS, tags=["lookup-lists"]
100+
)
101+
fastapi_app.include_router(
102+
heuristic_router, prefix=PREFIX_HEURISTIC, tags=["heuristic"]
103+
)
104+
fastapi_app.include_router(
105+
data_browser_router, prefix=PREFIX_DATA_BROWSER, tags=["data-browser"]
106+
)
107+
fastapi_app.include_router(labeling_router, prefix=PREFIX_LABELING, tags=["labeling"])
108+
fastapi_app.include_router(
109+
record_ide_router, prefix=PREFIX_RECORD_IDE, tags=["record-ide"]
110+
),
111+
fastapi_app.include_router(record_router, prefix=PREFIX_RECORD, tags=["record"]),
112+
fastapi_app.include_router(
113+
weak_supervision_router, prefix=PREFIX_WEAK_SUPERVISION, tags=["weak-supervision"]
114+
)
115+
fastapi_app.include_router(
116+
labeling_tasks_router, prefix=PREFIX_LABELING_TASKS, tags=["labeling-tasks"]
117+
)
33118

34119
routes = [
35120
Route(
@@ -64,6 +149,7 @@
64149
Route("/project", ProjectCreationFromWorkflow),
65150
Route("/is_managed", IsManagedRest),
66151
Route("/is_demo", IsDemoRest),
152+
Mount("/api", app=fastapi_app, name="REST API"),
67153
]
68154

69155
middleware = [Middleware(DatabaseSessionHandler)]

‎controller/auth/kratos.py‎

Lines changed: 54 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
from typing import Union, Any
1+
from typing import Union, Any, List, Optional, Dict
22
from requests import Response
33
import os
44
import requests
@@ -32,3 +32,56 @@ def resolve_user_name_by_id(user_id: str) -> str:
3232
if res.status_code == 200 and data["traits"]:
3333
return data["traits"]["name"]
3434
return None
35+
36+
37+
def resolve_all_user_ids(
38+
relevant_ids: List[str], as_list: bool = True
39+
) -> List[Dict[str, str]]:
40+
final = [] if as_list else {}
41+
for id in relevant_ids:
42+
r = requests.get(f"{KRATOS_ADMIN_URL}/identities/{id}").json()
43+
d = {
44+
"id": id,
45+
"mail": None,
46+
"firstName": None,
47+
"lastName": None,
48+
}
49+
if "traits" in r:
50+
traits = r["traits"]
51+
d["mail"] = traits["email"]
52+
d["firstName"] = traits["name"]["first"]
53+
d["lastName"] = traits["name"]["last"]
54+
if as_list:
55+
final.append(d)
56+
else:
57+
final[id] = d
58+
return final
59+
60+
61+
def expand_user_mail_name(
62+
users: List[Dict[str, str]], user_id_key="id"
63+
) -> List[Dict[str, str]]:
64+
final = []
65+
for user in users:
66+
r = requests.get(f"{KRATOS_ADMIN_URL}/identities/{user[user_id_key]}").json()
67+
d = {
68+
"mail": None,
69+
"firstName": None,
70+
"lastName": None,
71+
}
72+
if "traits" in r:
73+
traits = r["traits"]
74+
d["mail"] = traits["email"]
75+
d["firstName"] = traits["name"]["first"]
76+
d["lastName"] = traits["name"]["last"]
77+
user = {**user, **d}
78+
final.append(user)
79+
return final
80+
81+
82+
def resolve_user_name_and_email_by_id(user_id: str) -> dict:
83+
res: Response = requests.get("{}/identities/{}".format(KRATOS_ADMIN_URL, user_id))
84+
data: Any = res.json()
85+
if res.status_code == 200 and data["traits"]:
86+
return data["traits"]["name"], data["traits"]["email"]
87+
return None

‎controller/auth/manager.py‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
from typing import Any, Dict, List, Set
22

3+
from fastapi import Request
34
from graphene import ResolveInfo
45
from controller.misc import config_service
5-
from exceptions.exceptions import NotAllowedInDemoError
6+
from exceptions.exceptions import NotAllowedInDemoError, ProjectAccessError
67
import jwt
78
from graphql import GraphQLError
89
from controller.project import manager as project_manager
@@ -13,6 +14,8 @@
1314
from controller.misc import manager as misc_manager
1415
import sqlalchemy
1516

17+
DEV_USER_ID = "59e8dfca-ce56-44df-a8c7-5f05c61da499"
18+
1619

1720
def get_organization_id_by_info(info) -> Organization:
1821
organization: Organization = get_user_by_info(info).organization
@@ -22,7 +25,11 @@ def get_organization_id_by_info(info) -> Organization:
2225

2326

2427
def get_user_by_info(info) -> User:
25-
user_id: str = get_user_id_by_jwt_token(info.context["request"])
28+
request = info.context["request"]
29+
if request.url.hostname == "localhost" and request.url.port == 7051:
30+
user_id = DEV_USER_ID
31+
else:
32+
user_id: str = get_user_id_by_jwt_token(request)
2633
return user_manager.get_or_create_user(user_id)
2734

2835

@@ -61,6 +68,13 @@ def get_user_id_by_jwt_token(request) -> str:
6168
return claims["session"]["identity"]["id"]
6269

6370

71+
def check_project_access_dep(request: Request, project_id: str):
72+
if len(project_id) == 36:
73+
check_project_access(request.state.info, project_id)
74+
else:
75+
raise ProjectAccessError
76+
77+
6478
def check_project_access(info, project_id: str) -> None:
6579
organization_id: str = get_organization_id_by_info(info).id
6680
project: Project = project_manager.get_project_with_orga_id(
@@ -110,14 +124,14 @@ def check_is_admin(request: Any) -> bool:
110124
return False
111125

112126

113-
def check_demo_access(info: ResolveInfo) -> None:
127+
def check_demo_access(info: Any) -> None:
114128
if not check_is_admin(info.context["request"]) and config_service.get_config_value(
115129
"is_demo"
116130
):
117131
check_black_white(info)
118132

119133

120-
def check_black_white(info: ResolveInfo):
134+
def check_black_white(info: Any):
121135
black_white = misc_manager.get_black_white_demo()
122136
if str(info.parent_type) == "Mutation":
123137
if info.field_name not in black_white["mutations"]:

‎controller/comment/manager.py‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,10 @@ def get_comment(xftype: str, user_id: str, comment_id: str) -> CommentData:
4747
return comments.get_as_json(comment_id, user_id)
4848

4949

50+
def get_comment_by_comment_id(user_id: str, comment_id: str) -> CommentData:
51+
return comments.get_as_json(comment_id, user_id)
52+
53+
5054
def get_add_info(
5155
xftype: str,
5256
project_id: Optional[str] = None,
@@ -100,24 +104,24 @@ def update_comment(
100104
item = comments.get(comment_id)
101105

102106
if not item:
103-
raise ValueError(f"Can't find comment")
107+
raise ValueError("Can't find comment")
104108

105109
if user.role != enums.UserRoles.ENGINEER.value and user.id != item.created_by:
106-
raise ValueError(f"Can't update comment")
110+
raise ValueError("Can't update comment")
107111
comments.change(item, changes, with_commit=True)
108112
return item
109113

110114

111115
def delete_comment(comment_id: str, user_id: str) -> CommentData:
112116
item = comments.get(comment_id)
113117
if not item:
114-
raise ValueError(f"Can't find comment")
118+
raise ValueError("Can't find comment")
115119

116120
if (
117121
item.created_by != user_id
118122
and user_manager.get_user_role_by_id(user_id) != enums.UserRoles.ENGINEER.value
119123
):
120-
raise ValueError(f"Can't delete comment")
124+
raise ValueError("Can't delete comment")
121125
comments.remove(comment_id, with_commit=True)
122126

123127

‎controller/embedding/manager.py‎

Lines changed: 65 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,15 @@
99
from . import connector
1010
from .terms import TERMS_INFO
1111
from controller.model_provider import manager as model_manager
12-
from submodules.model.business_objects import attribute, embedding, agreement, general
12+
from submodules.model.business_objects import (
13+
attribute,
14+
embedding,
15+
agreement,
16+
general,
17+
project,
18+
)
19+
from submodules.model.util import sql_alchemy_to_dict
20+
from controller.embedding.connector import collection_on_qdrant
1321

1422

1523
def get_terms_info(
@@ -137,6 +145,62 @@ def get_embedding_name(
137145
return name
138146

139147

148+
EMBEDDING_SCHEMA_WHITELIST = [
149+
"id",
150+
"name",
151+
"custom",
152+
"type",
153+
"state",
154+
"progress",
155+
"dimension",
156+
"count",
157+
"platform",
158+
"model",
159+
"filter_attributes",
160+
"attribute_id",
161+
]
162+
163+
164+
def get_embedding_schema(project_id: str) -> List[Dict[str, Any]]:
165+
embeddings = embedding.get_all_embeddings_by_project_id(project_id)
166+
embedding_dict = sql_alchemy_to_dict(
167+
embeddings, column_whitelist=EMBEDDING_SCHEMA_WHITELIST
168+
)
169+
number_records = len(project.get(project_id).records)
170+
expanded_embeddings = []
171+
for embed in embedding_dict:
172+
count = embedding.get_tensor_count(embed["id"])
173+
onQdrant = collection_on_qdrant(project_id, embed["id"])
174+
175+
embedding_item = embedding.get_tensor(embed["id"])
176+
dimension = 0
177+
if embedding_item is not None:
178+
# distinguish between token and attribute embeddings
179+
if type(embedding_item.data[0]) is list:
180+
dimension = len(embedding_item.data[0])
181+
else:
182+
dimension = len(embedding_item.data)
183+
184+
if embed["state"] == "FINISHED":
185+
progress = 1
186+
elif embed["state"] == "INITIALIZING" or embed["state"] == "WAITING":
187+
progress = 0.0
188+
else:
189+
progress = min(
190+
0.1 + (count / number_records * 0.9),
191+
0.99,
192+
)
193+
expanded_embed = {
194+
**embed,
195+
"progress": progress,
196+
"count": count,
197+
"dimension": dimension,
198+
"onQdrant": onQdrant,
199+
}
200+
expanded_embeddings.append(expanded_embed)
201+
return {"id": project_id, "embeddings": expanded_embeddings}
202+
203+
140204
def recreate_embeddings(
141205
project_id: str, embedding_ids: Optional[List[str]] = None, user_id: str = None
142206
) -> None:

‎controller/knowledge_term/manager.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
EntityAlreadyExistsException,
77
EntityNotFoundException,
88
)
9-
from submodules.model.business_objects import general, knowledge_term, knowledge_base
9+
from submodules.model.business_objects import knowledge_term, knowledge_base
1010
from util.notification import create_notification
1111

1212

‎controller/labeling_task/manager.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
from submodules.model import LabelingTask
22
from .util import resolve_attribute_information
3-
from submodules.model.business_objects import labeling_task, general, attribute
3+
from submodules.model.business_objects import labeling_task, attribute
44

55

66
def get_labeling_task(project_id: str, labeling_task_id: str) -> LabelingTask:

0 commit comments

Comments
 (0)