Skip to content

Commit 2adcf2c

Browse files
ntkatholentkathole
authored
feat: Added recent visit logging api for registry server (feast-dev#5545)
Signed-off-by: ntkathole <[email protected]>
1 parent d095b96 commit 2adcf2c

File tree

17 files changed

+940
-14
lines changed

17 files changed

+940
-14
lines changed

‎docs/reference/feature-servers/registry-server.md

Lines changed: 187 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -946,4 +946,190 @@ Please refer the [page](./../registry/registry-permissions.md) for more details
946946

947947
## How to configure Authentication and Authorization ?
948948

949-
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.
949+
Please refer the [page](./../../../docs/getting-started/concepts/permission.md) for more details on how to configure authentication and authorization.
950+
951+
### Metrics
952+
953+
#### Get Resource Counts
954+
- **Endpoint**: `GET /api/v1/metrics/resource_counts`
955+
- **Description**: Retrieve counts of registry objects (entities, data sources, feature views, etc.) for a project or across all projects.
956+
- **Parameters**:
957+
- `project` (optional): Project name to filter resource counts (if not provided, returns counts for all projects)
958+
- **Examples**:
959+
```bash
960+
# Get counts for specific project
961+
curl -H "Authorization: Bearer <token>" \
962+
"http://localhost:6572/api/v1/metrics/resource_counts?project=my_project"
963+
964+
# Get counts for all projects
965+
curl -H "Authorization: Bearer <token>" \
966+
"http://localhost:6572/api/v1/metrics/resource_counts"
967+
```
968+
- **Response Example** (single project):
969+
```json
970+
{
971+
"project": "my_project",
972+
"counts": {
973+
"entities": 5,
974+
"dataSources": 3,
975+
"savedDatasets": 2,
976+
"features": 12,
977+
"featureViews": 4,
978+
"featureServices": 2
979+
}
980+
}
981+
```
982+
- **Response Example** (all projects):
983+
```json
984+
{
985+
"total": {
986+
"entities": 15,
987+
"dataSources": 8,
988+
"savedDatasets": 5,
989+
"features": 35,
990+
"featureViews": 12,
991+
"featureServices": 6
992+
},
993+
"perProject": {
994+
"project_a": {
995+
"entities": 5,
996+
"dataSources": 3,
997+
"savedDatasets": 2,
998+
"features": 12,
999+
"featureViews": 4,
1000+
"featureServices": 2
1001+
},
1002+
"project_b": {
1003+
"entities": 10,
1004+
"dataSources": 5,
1005+
"savedDatasets": 3,
1006+
"features": 23,
1007+
"featureViews": 8,
1008+
"featureServices": 4
1009+
}
1010+
}
1011+
}
1012+
```
1013+
1014+
#### Get Recently Visited Objects
1015+
- **Endpoint**: `GET /api/v1/metrics/recently_visited`
1016+
- **Description**: Retrieve the most recently visited registry objects for the authenticated user in a project.
1017+
- **Parameters**:
1018+
- `project` (optional): Project name to filter recent visits (defaults to current project)
1019+
- `object` (optional): Object type to filter recent visits (e.g., entities, features, feature_services)
1020+
- `page` (optional): Page number for pagination (starts from 1)
1021+
- `limit` (optional): Number of items per page (maximum 100)
1022+
- `sort_by` (optional): Field to sort by (e.g., timestamp, path, object)
1023+
- `sort_order` (optional): Sort order: "asc" or "desc" (default: "asc")
1024+
- **Examples**:
1025+
```bash
1026+
# Get all recent visits for a project
1027+
curl -H "Authorization: Bearer <token>" \
1028+
"http://localhost:6572/api/v1/metrics/recently_visited?project=my_project"
1029+
1030+
# Get recent visits with pagination
1031+
curl -H "Authorization: Bearer <token>" \
1032+
"http://localhost:6572/api/v1/metrics/recently_visited?project=my_project&page=1&limit=10"
1033+
1034+
# Get recent visits filtered by object type
1035+
curl -H "Authorization: Bearer <token>" \
1036+
"http://localhost:6572/api/v1/metrics/recently_visited?project=my_project&object=entities"
1037+
1038+
# Get recent visits sorted by timestamp descending
1039+
curl -H "Authorization: Bearer <token>" \
1040+
"http://localhost:6572/api/v1/metrics/recently_visited?project=my_project&sort_by=timestamp&sort_order=desc"
1041+
```
1042+
- **Response Example** (without pagination):
1043+
```json
1044+
{
1045+
"visits": [
1046+
{
1047+
"path": "/api/v1/entities/driver",
1048+
"timestamp": "2024-07-18T12:34:56.789Z",
1049+
"project": "my_project",
1050+
"user": "alice",
1051+
"object": "entities",
1052+
"object_name": "driver",
1053+
"method": "GET"
1054+
},
1055+
{
1056+
"path": "/api/v1/feature_services/user_service",
1057+
"timestamp": "2024-07-18T12:30:45.123Z",
1058+
"project": "my_project",
1059+
"user": "alice",
1060+
"object": "feature_services",
1061+
"object_name": "user_service",
1062+
"method": "GET"
1063+
}
1064+
],
1065+
"pagination": {
1066+
"totalCount": 2
1067+
}
1068+
}
1069+
```
1070+
- **Response Example** (with pagination):
1071+
```json
1072+
{
1073+
"visits": [
1074+
{
1075+
"path": "/api/v1/entities/driver",
1076+
"timestamp": "2024-07-18T12:34:56.789Z",
1077+
"project": "my_project",
1078+
"user": "alice",
1079+
"object": "entities",
1080+
"object_name": "driver",
1081+
"method": "GET"
1082+
}
1083+
],
1084+
"pagination": {
1085+
"page": 1,
1086+
"limit": 10,
1087+
"totalCount": 25,
1088+
"totalPages": 3,
1089+
"hasNext": true,
1090+
"hasPrevious": false
1091+
}
1092+
}
1093+
```
1094+
1095+
**Note**: Recent visits are automatically logged when users access registry objects via the REST API. The logging behavior can be configured through the `feature_server.recent_visit_logging` section in `feature_store.yaml` (see configuration section below).
1096+
1097+
---
1098+
1099+
## Registry Server Configuration: Recent Visit Logging
1100+
1101+
The registry server supports configuration of recent visit logging via the `feature_server` section in `feature_store.yaml`.
1102+
1103+
**Example:**
1104+
```yaml
1105+
feature_server:
1106+
type: local
1107+
recent_visit_logging:
1108+
limit: 100 # Number of recent visits to store per user
1109+
log_patterns:
1110+
- ".*/entities/(?!all$)[^/]+$"
1111+
- ".*/data_sources/(?!all$)[^/]+$"
1112+
- ".*/feature_views/(?!all$)[^/]+$"
1113+
- ".*/features/(?!all$)[^/]+$"
1114+
- ".*/feature_services/(?!all$)[^/]+$"
1115+
- ".*/saved_datasets/(?!all$)[^/]+$"
1116+
- ".*/custom_api/.*"
1117+
```
1118+
1119+
**Configuration Options:**
1120+
- **recent_visit_logging.limit**: Maximum number of recent visits to store per user (default: 100).
1121+
- **recent_visit_logging.log_patterns**: List of regex patterns for API paths to log as recent visits.
1122+
1123+
**Default Log Patterns:**
1124+
- `.*/entities/(?!all$)[^/]+$` - Individual entity endpoints
1125+
- `.*/data_sources/(?!all$)[^/]+$` - Individual data source endpoints
1126+
- `.*/feature_views/(?!all$)[^/]+$` - Individual feature view endpoints
1127+
- `.*/features/(?!all$)[^/]+$` - Individual feature endpoints
1128+
- `.*/feature_services/(?!all$)[^/]+$` - Individual feature service endpoints
1129+
- `.*/saved_datasets/(?!all$)[^/]+$` - Individual saved dataset endpoints
1130+
1131+
**Behavior:**
1132+
- Only requests matching one of the `log_patterns` will be tracked
1133+
- Only the most recent `limit` visits per user are stored
1134+
- Metrics endpoints (`/metrics/*`) are automatically excluded from logging to prevent circular references
1135+
- Visit data is stored per user and per project in the registry metadata

‎sdk/python/feast/api/registry/rest/__init__.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,13 @@
66
from feast.api.registry.rest.feature_views import get_feature_view_router
77
from feast.api.registry.rest.features import get_feature_router
88
from feast.api.registry.rest.lineage import get_lineage_router
9+
from feast.api.registry.rest.metrics import get_metrics_router
910
from feast.api.registry.rest.permissions import get_permission_router
1011
from feast.api.registry.rest.projects import get_project_router
1112
from feast.api.registry.rest.saved_datasets import get_saved_dataset_router
1213

1314

14-
def register_all_routes(app: FastAPI, grpc_handler):
15+
def register_all_routes(app: FastAPI, grpc_handler, server=None):
1516
app.include_router(get_entity_router(grpc_handler))
1617
app.include_router(get_data_source_router(grpc_handler))
1718
app.include_router(get_feature_service_router(grpc_handler))
@@ -21,3 +22,4 @@ def register_all_routes(app: FastAPI, grpc_handler):
2122
app.include_router(get_permission_router(grpc_handler))
2223
app.include_router(get_project_router(grpc_handler))
2324
app.include_router(get_saved_dataset_router(grpc_handler))
25+
app.include_router(get_metrics_router(grpc_handler, server))

‎sdk/python/feast/api/registry/rest/metrics.py

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
import json
2+
import logging
3+
from typing import Optional
4+
5+
from fastapi import APIRouter, Depends, Query, Request
6+
7+
from feast.api.registry.rest.rest_utils import (
8+
get_pagination_params,
9+
get_sorting_params,
10+
grpc_call,
11+
paginate_and_sort,
12+
)
13+
from feast.protos.feast.registry import RegistryServer_pb2
14+
15+
16+
def get_metrics_router(grpc_handler, server=None) -> APIRouter:
17+
logger = logging.getLogger(__name__)
18+
router = APIRouter()
19+
20+
@router.get("/metrics/resource_counts", tags=["Metrics"])
21+
async def resource_counts(
22+
project: Optional[str] = Query(
23+
None, description="Project name to filter resource counts"
24+
),
25+
):
26+
def count_resources_for_project(project_name: str):
27+
entities = grpc_call(
28+
grpc_handler.ListEntities,
29+
RegistryServer_pb2.ListEntitiesRequest(project=project_name),
30+
)
31+
data_sources = grpc_call(
32+
grpc_handler.ListDataSources,
33+
RegistryServer_pb2.ListDataSourcesRequest(project=project_name),
34+
)
35+
try:
36+
saved_datasets = grpc_call(
37+
grpc_handler.ListSavedDatasets,
38+
RegistryServer_pb2.ListSavedDatasetsRequest(project=project_name),
39+
)
40+
except Exception:
41+
saved_datasets = {"savedDatasets": []}
42+
try:
43+
features = grpc_call(
44+
grpc_handler.ListFeatures,
45+
RegistryServer_pb2.ListFeaturesRequest(project=project_name),
46+
)
47+
except Exception:
48+
features = {"features": []}
49+
try:
50+
feature_views = grpc_call(
51+
grpc_handler.ListFeatureViews,
52+
RegistryServer_pb2.ListFeatureViewsRequest(project=project_name),
53+
)
54+
except Exception:
55+
feature_views = {"featureViews": []}
56+
try:
57+
feature_services = grpc_call(
58+
grpc_handler.ListFeatureServices,
59+
RegistryServer_pb2.ListFeatureServicesRequest(project=project_name),
60+
)
61+
except Exception:
62+
feature_services = {"featureServices": []}
63+
return {
64+
"entities": len(entities.get("entities", [])),
65+
"dataSources": len(data_sources.get("dataSources", [])),
66+
"savedDatasets": len(saved_datasets.get("savedDatasets", [])),
67+
"features": len(features.get("features", [])),
68+
"featureViews": len(feature_views.get("featureViews", [])),
69+
"featureServices": len(feature_services.get("featureServices", [])),
70+
}
71+
72+
if project:
73+
counts = count_resources_for_project(project)
74+
return {"project": project, "counts": counts}
75+
else:
76+
# List all projects via gRPC
77+
projects_resp = grpc_call(
78+
grpc_handler.ListProjects, RegistryServer_pb2.ListProjectsRequest()
79+
)
80+
all_projects = [
81+
p["spec"]["name"] for p in projects_resp.get("projects", [])
82+
]
83+
all_counts = {}
84+
total_counts = {
85+
"entities": 0,
86+
"dataSources": 0,
87+
"savedDatasets": 0,
88+
"features": 0,
89+
"featureViews": 0,
90+
"featureServices": 0,
91+
}
92+
for project_name in all_projects:
93+
counts = count_resources_for_project(project_name)
94+
all_counts[project_name] = counts
95+
for k in total_counts:
96+
total_counts[k] += counts[k]
97+
return {"total": total_counts, "perProject": all_counts}
98+
99+
@router.get("/metrics/recently_visited", tags=["Metrics"])
100+
async def recently_visited(
101+
request: Request,
102+
project: Optional[str] = Query(
103+
None, description="Project name to filter recent visits"
104+
),
105+
object_type: Optional[str] = Query(
106+
None,
107+
alias="object",
108+
description="Object type to filter recent visits (e.g., entities, features)",
109+
),
110+
pagination_params: dict = Depends(get_pagination_params),
111+
sorting_params: dict = Depends(get_sorting_params),
112+
):
113+
user = None
114+
if hasattr(request.state, "user"):
115+
user = getattr(request.state, "user", None)
116+
if not user:
117+
user = "anonymous"
118+
project_val = project or (server.store.project if server else None)
119+
key = f"recently_visited_{user}"
120+
logger.info(
121+
f"[/metrics/recently_visited] Project: {project_val}, Key: {key}, Object: {object_type}"
122+
)
123+
try:
124+
visits_json = (
125+
server.registry.get_project_metadata(project_val, key)
126+
if server
127+
else None
128+
)
129+
visits = json.loads(visits_json) if visits_json else []
130+
except Exception:
131+
visits = []
132+
if object_type:
133+
visits = [v for v in visits if v.get("object") == object_type]
134+
135+
server_limit = getattr(server, "recent_visits_limit", 100) if server else 100
136+
visits = visits[-server_limit:]
137+
138+
page = pagination_params.get("page", 0)
139+
limit = pagination_params.get("limit", 0)
140+
sort_by = sorting_params.get("sort_by")
141+
sort_order = sorting_params.get("sort_order", "asc")
142+
143+
if page == 0 and limit == 0:
144+
if sort_by:
145+
visits = sorted(
146+
visits,
147+
key=lambda x: x.get(sort_by, ""),
148+
reverse=(sort_order == "desc"),
149+
)
150+
return {"visits": visits, "pagination": {"totalCount": len(visits)}}
151+
else:
152+
if page == 0:
153+
page = 1
154+
if limit == 0:
155+
limit = 50
156+
paged_visits, pagination = paginate_and_sort(
157+
visits, page, limit, sort_by, sort_order
158+
)
159+
return {
160+
"visits": paged_visits,
161+
"pagination": pagination,
162+
}
163+
164+
return router

0 commit comments

Comments
 (0)