Skip to content

Commit b7a3422

Browse files
feat: backend streaming support and storage v2 optimizations (#351)
* feat: backend streaming support and storage v2 optimizations * feat: harden storage v2 and improve profile placeholder visibility --------- Co-authored-by: Dave Augustus <[email protected]>
1 parent 99b1e2a commit b7a3422

File tree

3 files changed

+169
-32
lines changed

3 files changed

+169
-32
lines changed

documents/api_usage_report.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Definitive API Integration Status Report (Highly Detailed)
2+
3+
**Date**: 2026-01-25 07:55 PM
4+
**Status**: 100% Manually Traced & Verified (Updated with Storage V2)
5+
**Scope**: All 9 Gateway Internal Services (Auth, User, Blog, Notification, Storage v1/v2, AI, Admin, Systems, Activity)
6+
7+
---
8+
9+
## 1. INTEGRATED APIs (Confirmed in Frontend)
10+
11+
These APIs are actively used in the frontend codebase. They are grouped by their functional version (V1 vs V2) and include their specific transport method.
12+
13+
### 🟢 Blog & News Engine (V1 & V2)
14+
| Version | Method | Endpoint Path | Description | Integration Detail |
15+
|:---:|:---:|:---|:---|:---|
16+
| **V2** | **WS** | `/blog/draft/:blog_id` | **Real-time drafting** | Used in `create` & `edit` pages for auto-save/sync. |
17+
| **V2** | **GET** | `/blog/meta-feed` | Primary feed engine | Combined metadata stream for the home page. |
18+
| **V2** | **GET** | `/blog/feed` | Latest updates | High-performance stream of recent blogs. |
19+
| **V2** | **GET** | `/blog/:blog_id` | Blog document fetch | Used in individual blog view pages (`[slug]`). |
20+
| **V2** | **GET** | `/blog/in-my-draft` | Personal draft list | Library dashboard integration (User's own drafts). |
21+
| **V2** | **GET** | `/blog/in-my-bookmark` | Saved blogs list | Library dashboard integration (User's bookmarks). |
22+
| **V2** | **GET** | `/blog/following` | Followed feed | Personalized stream of blogs from followed authors. |
23+
| **V2** | **GET** | `/blog/user/:username` | Author blog list | Public profile integration (List of author's blogs). |
24+
| **V2** | **GET** | `/blog/search` | Search engine | Global blog search with limit/offset support. |
25+
| **V1** | **POST** | `/blog/publish/:blog_id` | Publication trigger | Final step in the creation/editing flow. |
26+
| **V2** | **POST** | `/blog/to-draft/:blog_id` | Revert to draft | Used to pull a published blog back into drafting. |
27+
28+
### 🟢 Storage & Assets (V1 Legacy Stack)
29+
*Note: The frontend is currently locked to the V1 File Service APIs.*
30+
| Version | Method | Endpoint Path | Description | Integration Detail |
31+
|:---:|:---:|:---|:---|:---|
32+
| **V1** | **POST** | `/files/post/:id` | Blog image upload | EditorJS integration for inline images. |
33+
| **V1** | **GET** | `/files/post/:id/:fileName` | Serve blog assets | Dynamic image rendering in published blogs. |
34+
| **V1** | **POST** | `/files/profile/:id/profile` | Profile pic upload | Update profile dialog integration. |
35+
| **V1** | **GET** | `/files/profile/:id/profile` | Serve avatar | Global layout sidebar and public profile usage. |
36+
| **V1.1** | **GET** | `/files/profile/:id/profile` | Avatar stream | High-performance binary stream for profile images. |
37+
38+
### 🟢 Modern Storage Stack (V2 MinIO - Migration Active)
39+
*The frontend has successfully migrated core asset handling to Storage V2.*
40+
| Version | Method | Endpoint Path | Description | Integration Detail |
41+
|:---:|:---:|:---|:---|:---|
42+
| **V2** | **POST** | `/storage/posts/:id` | Blog asset upload | Integrated in EditorJS (Images, Videos, PDFs). |
43+
| **V2** | **GET** | `/storage/posts/:id/:file/url` | Asset delivery | Used for dynamic resolution in Editor and Reader. |
44+
| **V2** | **DELETE** | `/storage/posts/:id/:file` | Asset cleanup | **Automatic deletion hook** on block removal. |
45+
| **V2** | **POST** | `/storage/profiles/:id/profile`| Profile upload | Migrated in `UpdateProfileDialog`. |
46+
| **V2** | **GET** | `/storage/profiles/:id/profile/url`| Avatar delivery | Migrated in `useProfileImage` hook. |
47+
| **V2** | **DELETE** | `/storage/profiles/:id/profile`| Profile delete | **Active** in `UpdateProfileDialog` (Confirmation UI). |
48+
49+
### 🟢 User & Profile Service (V1)
50+
| Version | Method | Endpoint Path | Description | Integration Detail |
51+
|:---:|:---:|:---|:---|:---|
52+
| **V1** | **GET** | `/user/public/:id` | Username profile | Fetch public data by username handle. |
53+
| **V1** | **GET** | `/user/public/account/:acc_id` | Account ID profile | Fetch public data via internal account ID. |
54+
| **V1** | **GET** | `/user/connection-count/:user` | Stats integration | Follower and following count for profiles. |
55+
| **V1** | **POST** | `/user/follow/:username` | Follow action | Confirm/Toggle follower status. |
56+
| **V1** | **POST** | `/user/unfollow/:username` | Unfollow action | Remove follower status. |
57+
| **V1** | **PUT/PATCH** | `/user/:id` | Profile settings | Full or partial (patch) profile updates. |
58+
| **V1** | **GET** | `/user/topics` | Topic registry | Fetch all valid tags for global exploration. |
59+
| **V1** | **POST** | `/user/topics` | Topic creation | Author-defined topic registration. |
60+
| **V1** | **GET** | `/user/activities/:user` | Activity log | Integrated public activity history timeline. |
61+
62+
### 🟢 Authentication Service (V1)
63+
| Version | Method | Endpoint Path | Description | Integration Detail |
64+
|:---:|:---:|:---|:---|:---|
65+
| **V1** | **POST** | `/auth/login` | Login | Core authentication flow. |
66+
| **V1** | **POST** | `/auth/register` | Signup | New user registration flow. |
67+
| **V1** | **GET** | `/auth/validate-session` | Auth check | Initial app load context verification. |
68+
| **V1** | **POST** | `/auth/refresh` | Token sync | Automatic silent refresh via Axios interceptors. |
69+
| **V1** | **GET** | `/auth/ws-token` | WebSocket Auth | Obtaining one-time ticket for WS connections. |
70+
71+
### 🟢 Notification & System (V1)
72+
| Version | Method | Endpoint Path | Description | Integration Detail |
73+
|:---:|:---:|:---|:---|:---|
74+
| **V1** | **WS** | `/notification/ws-notification` | **Global Events** | Real-time stream confirmed in `WSNotificationDropdown`. |
75+
| **V1** | **GET** | `/notification/notifications` | Feed fetch | Manual inbox retrieval for the library inbox. |
76+
| **V1** | **POST** | `/contact` | Lead generation | Confirmed in the Public Contact Us support form. |
77+
78+
---
79+
80+
## 2. PENDING APIs (Backend-Only / Not in Frontend)
81+
82+
These APIs represent implemented backend features that are not yet exposed or utilized in the frontend UI.
83+
84+
| **V2** | **HEAD** | `/storage/posts/:id/:file` | Asset metadata | Pending Frontend Migration. |
85+
| **V2** | **LIST** | `/storage/posts/:id` | Folder listing | Pending Frontend Migration. |
86+
| **V2** | **GET** | `/storage/posts/:id/:fileName/meta` | Deep metadata | Pending Metadata rendering (Blurhash/Dimensions). |
87+
88+
### 🔴 Secure Administrative Tools (Admin Service)
89+
| Version | Method | Endpoint Path | Description | Status |
90+
|:---:|:---:|:---|:---|:---|
91+
| **V1** | **GET** | `/admin/health` | Service status | Backend Cluster Monitoring only. |
92+
| **V1** | **DELETE** | `/admin/users/flag` | User banning | Admin Panel implementation pending. |
93+
| **V1** | **POST** | `/admin/backup/trigger`| DR recovery | DevOps/Internal CLI usage only. |
94+
95+
### 🔴 AI & Collaborative Intelligence
96+
| Version | Method | Endpoint Path | Description | Status |
97+
|:---:|:---:|:---|:---|:---|
98+
| **V1** | **GET** | `/recommendations/*` | AI Suggestions | Recommendation Engine implementation pending. |
99+
100+
### 🔴 Advanced Blog & User Management (Waitlist)
101+
| Version | Method | Endpoint Path | Description | Status |
102+
|:---:|:---:|:---|:---|:---|
103+
| **V1** | **POST** | `/user/invite/:blog_id` | Co-author invite | Collaborative writing feature pending. |
104+
| **V1** | **POST** | `/blog/archive/:blog_id` | Soft delete | Archive UI integration pending. |
105+
| **V2** | **GET** | `/user/active-users` | Real-time stats | Admin Dashboard feature pending. |
106+
| **V2** | **GET** | `/blog/:blog_id/stats` | Deep analytics | Premium stats view integration pending. |

microservices/the_monkeys_gateway/internal/storage_v2/routes.go

Lines changed: 61 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -246,29 +246,37 @@ func (s *Service) UploadPostFile(ctx *gin.Context) {
246246
objectName := "posts/" + blogID + "/" + fname
247247
contentType := fileHeader.Header.Get("Content-Type")
248248

249-
// Read into memory once so we can compute image metadata (BlurHash) and upload
250-
data, err := io.ReadAll(file)
251-
if err != nil {
252-
ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"message": "read failed"})
253-
return
254-
}
255-
reader := bytes.NewReader(data)
256-
257249
opts := minio.PutObjectOptions{
258250
ContentType: contentType,
259-
CacheControl: "public, max-age=31536000, immutable",
251+
CacheControl: "public, max-age=31536000",
260252
}
261-
if strings.HasPrefix(strings.ToLower(contentType), "image/") {
262-
if hash, w, h, ok := s.computeImageMetadata(contentType, data); ok {
263-
opts.UserMetadata = map[string]string{
264-
"x-blurhash": hash,
265-
"x-width": strconv.Itoa(w),
266-
"x-height": strconv.Itoa(h),
253+
254+
// Read a small prefix for image metadata if it's an image
255+
// 5MB limit for metadata processing to keep memory low
256+
const metadataLimit = 5 * 1024 * 1024
257+
var finalReader io.Reader = file
258+
var objectSize int64 = fileHeader.Size
259+
260+
if strings.HasPrefix(strings.ToLower(contentType), "image/") && fileHeader.Size <= metadataLimit {
261+
// Read into memory ONLY if small enough for metadata
262+
data, err := io.ReadAll(file)
263+
if err == nil {
264+
if hash, w, h, ok := s.computeImageMetadata(contentType, data); ok {
265+
opts.UserMetadata = map[string]string{
266+
"x-blurhash": hash,
267+
"x-width": strconv.Itoa(w),
268+
"x-height": strconv.Itoa(h),
269+
}
267270
}
271+
finalReader = bytes.NewReader(data)
272+
objectSize = int64(len(data))
273+
} else {
274+
// Fallback to streaming if read fails
275+
_, _ = file.Seek(0, io.SeekStart)
268276
}
269277
}
270278

271-
info, err := s.mc.PutObject(ctx.Request.Context(), s.bucket, objectName, reader, int64(len(data)), opts)
279+
info, err := s.mc.PutObject(ctx.Request.Context(), s.bucket, objectName, finalReader, objectSize, opts)
272280
if err != nil {
273281
s.log.Errorf("minio PutObject error: %v", err)
274282
ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"message": "upload failed"})
@@ -396,7 +404,7 @@ func (s *Service) UpdatePostFile(ctx *gin.Context) {
396404

397405
opts := minio.PutObjectOptions{
398406
ContentType: contentType,
399-
CacheControl: "public, max-age=31536000, immutable",
407+
CacheControl: "public, max-age=31536000",
400408
}
401409
if strings.HasPrefix(strings.ToLower(contentType), "image/") {
402410
if hash, w, h, ok := s.computeImageMetadata(contentType, data); ok {
@@ -473,6 +481,11 @@ func (s *Service) GetPostFile(ctx *gin.Context) {
473481
}
474482
}
475483

484+
// For PDFs, ensure inline display in iframes
485+
if strings.Contains(strings.ToLower(stat.ContentType), "application/pdf") {
486+
ctx.Header("Content-Disposition", "inline")
487+
}
488+
476489
// Stream body
477490
if _, err := io.Copy(ctx.Writer, obj); err != nil {
478491
s.log.Errorf("stream write error: %v", err)
@@ -535,28 +548,34 @@ func (s *Service) UploadProfileImage(ctx *gin.Context) {
535548
objectName := "profiles/" + userID + "/profile" // single canonical key for profile image
536549
contentType := fileHeader.Header.Get("Content-Type")
537550

538-
data, err := io.ReadAll(file)
539-
if err != nil {
540-
ctx.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"message": "read failed"})
541-
return
542-
}
543-
reader := bytes.NewReader(data)
551+
// Streaming upload
552+
const metadataLimit = 5 * 1024 * 1024
553+
var finalReader io.Reader = file
554+
var objectSize int64 = fileHeader.Size
544555

545556
opts := minio.PutObjectOptions{
546557
ContentType: contentType,
547558
CacheControl: "public, max-age=31536000, immutable",
548559
}
549-
if strings.HasPrefix(strings.ToLower(contentType), "image/") {
550-
if hash, w, h, ok := s.computeImageMetadata(contentType, data); ok {
551-
opts.UserMetadata = map[string]string{
552-
"x-blurhash": hash,
553-
"x-width": strconv.Itoa(w),
554-
"x-height": strconv.Itoa(h),
560+
561+
if strings.HasPrefix(strings.ToLower(contentType), "image/") && fileHeader.Size <= metadataLimit {
562+
data, err := io.ReadAll(file)
563+
if err == nil {
564+
if hash, w, h, ok := s.computeImageMetadata(contentType, data); ok {
565+
opts.UserMetadata = map[string]string{
566+
"x-blurhash": hash,
567+
"x-width": strconv.Itoa(w),
568+
"x-height": strconv.Itoa(h),
569+
}
555570
}
571+
finalReader = bytes.NewReader(data)
572+
objectSize = int64(len(data))
573+
} else {
574+
_, _ = file.Seek(0, io.SeekStart)
556575
}
557576
}
558577

559-
info, err := s.mc.PutObject(ctx.Request.Context(), s.bucket, objectName, reader, int64(len(data)), opts)
578+
info, err := s.mc.PutObject(ctx.Request.Context(), s.bucket, objectName, finalReader, objectSize, opts)
560579
if err != nil {
561580
s.log.Errorf("minio PutObject error: %v", err)
562581
ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"message": "upload failed"})
@@ -789,6 +808,12 @@ func (s *Service) GetPostFileURL(ctx *gin.Context) {
789808
}
790809
}
791810

811+
// Check existence first
812+
if _, err := s.mc.StatObject(ctx.Request.Context(), s.bucket, objectName, minio.StatObjectOptions{}); err != nil {
813+
ctx.AbortWithStatusJSON(http.StatusNotFound, gin.H{"message": "file not found"})
814+
return
815+
}
816+
792817
urlStr, err := s.presignedOrCDNURL(ctx.Request.Context(), objectName, time.Duration(expires)*time.Second)
793818
if err != nil {
794819
ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"message": "could not generate url"})
@@ -817,6 +842,12 @@ func (s *Service) GetProfileURL(ctx *gin.Context) {
817842
}
818843
}
819844

845+
// Check existence first
846+
if _, err := s.mc.StatObject(ctx.Request.Context(), s.bucket, objectName, minio.StatObjectOptions{}); err != nil {
847+
ctx.AbortWithStatusJSON(http.StatusNotFound, gin.H{"message": "profile not found"})
848+
return
849+
}
850+
820851
urlStr, err := s.presignedOrCDNURL(ctx.Request.Context(), objectName, time.Duration(expires)*time.Second)
821852
if err != nil {
822853
ctx.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"message": "could not generate url"})

microservices/the_monkeys_gateway/main.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,11 @@ func main() {
6464
server.router.Use(gin.Recovery())
6565
// retain default gin logger? use custom zap middleware later
6666
// server.router.Use(gin.Logger())
67-
server.router.MaxMultipartMemory = 8 << 20
67+
server.router.MaxMultipartMemory = 100 << 20
6868

6969
// Apply security middleware
7070
server.router.Use(secure.New(secure.Config{
71-
FrameDeny: true,
71+
FrameDeny: false,
7272
ContentTypeNosniff: true,
7373
BrowserXssFilter: true,
7474
ContentSecurityPolicy: "default-src 'self';", // Customize as needed

0 commit comments

Comments
 (0)