Skip to content

Commit 04c8b13

Browse files
Implement maxAge fast scraping parameter (#73)
* Implement maxAge fast scraping parameter - Add maxAge parameter to SCRAPE_TOOL inputSchema with proper description - Update tool description to highlight 500% performance improvement - Add test coverage for maxAge parameter functionality - Parameter enables cached scraping for faster response times Fixes #69 Co-Authored-By: Nick <nicolascamara29@gmail.com> * Update package-lock.json after npm install Co-Authored-By: Nick <nicolascamara29@gmail.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Nick <nicolascamara29@gmail.com>
1 parent d8e8d28 commit 04c8b13

File tree

2 files changed

+42
-1
lines changed

2 files changed

+42
-1
lines changed

src/index.test.ts

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,41 @@ describe('Firecrawl Tool Tests', () => {
139139
});
140140
});
141141

142+
// Test scrape with maxAge parameter
143+
test('should handle scrape request with maxAge parameter', async () => {
144+
const url = 'https://example.com';
145+
const options = { formats: ['markdown'], maxAge: 3600000 };
146+
147+
const mockResponse: ScrapeResponse = {
148+
success: true,
149+
markdown: '# Test Content',
150+
html: undefined,
151+
rawHtml: undefined,
152+
url: 'https://example.com',
153+
actions: undefined as never,
154+
};
155+
156+
mockClient.scrapeUrl.mockResolvedValueOnce(mockResponse);
157+
158+
const response = await requestHandler({
159+
method: 'call_tool',
160+
params: {
161+
name: 'firecrawl_scrape',
162+
arguments: { url, ...options },
163+
},
164+
});
165+
166+
expect(response).toEqual({
167+
content: [{ type: 'text', text: '# Test Content' }],
168+
isError: false,
169+
});
170+
expect(mockClient.scrapeUrl).toHaveBeenCalledWith(url, {
171+
formats: ['markdown'],
172+
maxAge: 3600000,
173+
url,
174+
});
175+
});
176+
142177
// Test batch scrape functionality
143178
test('should handle batch scrape request', async () => {
144179
const urls = ['https://example.com'];

src/index.ts

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,12 @@ Scrape content from a single URL with advanced options.
3636
"name": "firecrawl_scrape",
3737
"arguments": {
3838
"url": "https://example.com",
39-
"formats": ["markdown"]
39+
"formats": ["markdown"],
40+
"maxAge": 3600000
4041
}
4142
}
4243
\`\`\`
44+
**Performance:** Add maxAge parameter for 500% faster scrapes using cached data.
4345
**Returns:** Markdown, HTML, or other formats as specified.
4446
`,
4547
inputSchema: {
@@ -188,6 +190,10 @@ Scrape content from a single URL with advanced options.
188190
},
189191
description: 'Location settings for scraping',
190192
},
193+
maxAge: {
194+
type: 'number',
195+
description: 'Maximum age in milliseconds for cached content. Use cached data if available and younger than maxAge, otherwise scrape fresh. Enables 500% faster scrapes for recently cached pages. Default: 0 (always scrape fresh)',
196+
},
191197
},
192198
required: ['url'],
193199
},

0 commit comments

Comments
 (0)