Skip to content

Commit ea021d5

Browse files
Sahil-SimformSahil-Simform
committed
Added netlify cron jobs and storage
Signed-off-by: Sahil-simform <[email protected]>
1 parent 8419aad commit ea021d5

File tree

540 files changed

+62856
-3
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

540 files changed

+62856
-3
lines changed

‎.vscode/settings.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"deno.enable": true,
3+
"deno.enablePaths": [
4+
"netlify/edge-functions"
5+
],
6+
"deno.unstable": true,
7+
"deno.importMap": ".netlify/edge-functions-import-map.json",
8+
"deno.path": "/Users/sahil.t/Library/Preferences/netlify/deno-cli/deno"
9+
}

‎README.md‎

Lines changed: 231 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,231 @@
1+
# Netlify Functions for Package Data Collection
2+
3+
This project contains Netlify Functions and Edge Functions to periodically collect package data from pub.dev and serve it to your frontend.
4+
5+
## System Architecture
6+
7+
The system consists of three main components:
8+
9+
1. **Scheduled Function (`fetch-package-data.js`)**: Runs every 5 minutes to collect package data from pub.dev and store it in Netlify's KV store.
10+
2. **Package Management API (`manage-packages.js`)**: A Netlify Function for adding, removing, or updating the list of packages to track.
11+
3. **Data Access API (`get-package-data.js`)**: An Edge Function that serves cached package data to your frontend.
12+
13+
## Dependencies
14+
15+
The following Node.js packages are required:
16+
17+
```json
18+
{
19+
"dependencies": {
20+
"node-fetch": "^2.7.0",
21+
"@netlify/functions": "^2.4.1",
22+
"@netlify/blobs": "^6.5.0",
23+
"axios": "^1.6.2"
24+
}
25+
}
26+
```
27+
28+
Install them with:
29+
30+
```bash
31+
npm install
32+
```
33+
34+
## API Documentation
35+
36+
### 1. Package Management API
37+
38+
**Endpoint:** `/.netlify/functions/manage-packages`
39+
40+
#### Get the current list of tracked packages
41+
42+
**Method:** GET
43+
**Example:**
44+
```bash
45+
curl https://your-site.netlify.app/.netlify/functions/manage-packages
46+
```
47+
48+
**Response:**
49+
```json
50+
{
51+
"packages": ["flutter", "http", "provider", "shared_preferences"]
52+
}
53+
```
54+
55+
#### Add packages
56+
57+
**Method:** POST
58+
**Body:**
59+
```json
60+
{
61+
"action": "add",
62+
"package": "flutter_bloc"
63+
}
64+
```
65+
66+
or add multiple packages:
67+
68+
```json
69+
{
70+
"action": "add",
71+
"packages": ["flutter_bloc", "get_it", "dio"]
72+
}
73+
```
74+
75+
**Example:**
76+
```bash
77+
curl -X POST https://your-site.netlify.app/.netlify/functions/manage-packages \
78+
-H "Content-Type: application/json" \
79+
-d '{"action": "add", "package": "equatable"}'
80+
```
81+
82+
**Response:**
83+
```json
84+
{
85+
"message": "Packages added successfully",
86+
"packages": ["flutter", "http", "provider", "shared_preferences", "equatable"]
87+
}
88+
```
89+
90+
#### Remove packages
91+
92+
**Method:** POST
93+
**Body:**
94+
```json
95+
{
96+
"action": "remove",
97+
"package": "http"
98+
}
99+
```
100+
101+
or remove multiple packages:
102+
103+
```json
104+
{
105+
"action": "remove",
106+
"packages": ["provider", "http"]
107+
}
108+
```
109+
110+
**Example:**
111+
```bash
112+
curl -X POST https://your-site.netlify.app/.netlify/functions/manage-packages \
113+
-H "Content-Type: application/json" \
114+
-d '{"action": "remove", "package": "http"}'
115+
```
116+
117+
**Response:**
118+
```json
119+
{
120+
"message": "Packages removed successfully",
121+
"packages": ["flutter", "provider", "shared_preferences"]
122+
}
123+
```
124+
125+
#### Set the entire package list
126+
127+
**Method:** POST
128+
**Body:**
129+
```json
130+
{
131+
"action": "set",
132+
"packages": ["flutter", "flutter_bloc", "equatable", "dio", "get_it"]
133+
}
134+
```
135+
136+
**Example:**
137+
```bash
138+
curl -X POST https://your-site.netlify.app/.netlify/functions/manage-packages \
139+
-H "Content-Type: application/json" \
140+
-d '{"action":"set","packages":["flutter","flutter_bloc"]}'
141+
```
142+
143+
**Response:**
144+
```json
145+
{
146+
"message": "Packages set successfully",
147+
"packages": ["flutter", "flutter_bloc"]
148+
}
149+
```
150+
151+
### 2. Package Data API
152+
153+
**Endpoint:** `/api/package-data`
154+
155+
#### Get data for all tracked packages
156+
157+
**Method:** GET
158+
**Example:**
159+
```bash
160+
curl https://your-site.netlify.app/api/package-data
161+
```
162+
163+
**Response:**
164+
```json
165+
{
166+
"data": {
167+
"flutter": {
168+
"grantedPoints": 150,
169+
"maxPoints": 160,
170+
"likeCount": 2817,
171+
"downloadCount30Days": 156194,
172+
"tags": ["publisher:simform.com", "sdk:flutter", "..."],
173+
"lastUpdated": "2025-04-24T20:09:55.534241"
174+
},
175+
"http": {
176+
"...": "..."
177+
}
178+
},
179+
"lastUpdated": "2025-05-01T12:00:00.000Z",
180+
"environment": "production"
181+
}
182+
```
183+
184+
#### Get data for a specific package
185+
186+
**Method:** GET
187+
**Example:**
188+
```bash
189+
curl https://your-site.netlify.app/api/package-data?package=flutter
190+
```
191+
192+
**Response:**
193+
```json
194+
{
195+
"data": {
196+
"grantedPoints": 150,
197+
"maxPoints": 160,
198+
"likeCount": 2817,
199+
"downloadCount30Days": 156194,
200+
"tags": ["publisher:simform.com", "sdk:flutter", "..."],
201+
"lastUpdated": "2025-04-24T20:09:55.534241"
202+
},
203+
"lastUpdated": "2025-05-01T12:00:00.000Z",
204+
"environment": "production"
205+
}
206+
```
207+
208+
## Local Testing
209+
210+
To test your functions locally:
211+
212+
1. Install the Netlify CLI: `npm install -g netlify-cli`
213+
2. Run the development server: `netlify dev`
214+
3. Use the test script to verify the scheduled function: `node test-scheduled-function.js`
215+
216+
## Scheduled Function Testing
217+
218+
The scheduled function is configured to run every 5 minutes in production. To test it locally, use the test script that will simulate the scheduled execution.
219+
220+
## Deployment
221+
222+
When deployed to Netlify, make sure to:
223+
224+
1. Enable Netlify Blobs and KV Store for your site
225+
2. Verify that the scheduled function is running by checking the function logs
226+
227+
## Troubleshooting
228+
229+
- If the scheduled function isn't running, check the function logs in the Netlify dashboard
230+
- If data isn't being stored, verify that Netlify Blobs is properly configured for your site
231+
- For local development issues, make sure you have the Netlify CLI installed and are running `netlify dev`

‎netlify.toml‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,21 @@
11
[build]
2-
functions = "netlify_functions"
2+
functions = "netlify_functions"
3+
4+
# Edge function route configuration
5+
[[edge_functions]]
6+
path = "/api/package-data"
7+
function = "get-package-data"
8+
9+
# Enable scheduled functions
10+
[functions]
11+
node_bundler = "esbuild"
12+
external_node_modules = ["@netlify/functions", "@netlify/blobs", "axios"]
13+
14+
# Scheduled function configuration
15+
[functions.fetch-package-data]
16+
schedule = "*/5 * * * *"
17+
18+
# KV Store configuration
19+
[[kv_namespaces]]
20+
binding = "KV"
21+
namespace = "package-scores"

0 commit comments

Comments
 (0)