New Components - oxylabs #17218
New Components - oxylabs #17218
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 3 Skipped Deployments
|
WalkthroughSeveral new modules and actions were added to the Oxylabs integration, including actions for scraping URLs and queries, creating proxy sessions, and scheduling jobs. A new source for detecting completed scheduled runs was introduced. Utility and constants modules were added, and the main app implementation was migrated from TypeScript to JavaScript. The Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Action (Scrape URL)
participant Oxylabs App
participant Oxylabs API
User->>Action (Scrape URL): Provide URL, source, geoLocation
Action (Scrape URL)->>Oxylabs App: scrape({ url, source, geoLocation })
Oxylabs App->>Oxylabs API: POST /queries with params
Oxylabs API-->>Oxylabs App: Scrape response
Oxylabs App-->>Action (Scrape URL): Return response
Action (Scrape URL)-->>User: Return scrape result
sequenceDiagram
participant User
participant Action (Create Schedule)
participant Oxylabs App
participant Oxylabs API
User->>Action (Create Schedule): Provide cron, items, endTime
Action (Create Schedule)->>Oxylabs App: createSchedule({ cron, items, endTime })
Oxylabs App->>Oxylabs API: POST /schedules with schedule data
Oxylabs API-->>Oxylabs App: Schedule creation response
Oxylabs App-->>Action (Create Schedule): Return response
Action (Create Schedule)-->>User: Return schedule info
sequenceDiagram
participant Source (New Scheduled Run Completed)
participant Oxylabs App
participant Oxylabs API
participant Event Stream
Source (New Scheduled Run Completed)->>Oxylabs App: getRunsInfo({ scheduleId })
Oxylabs App->>Oxylabs API: GET /schedules/{scheduleId}/runs
Oxylabs API-->>Oxylabs App: List of runs
Oxylabs App-->>Source (New Scheduled Run Completed): Return runs
Source (New Scheduled Run Completed)->>Event Stream: Emit new run events
Assessment against linked issues
Assessment against linked issues: Out-of-scope changes
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
components/oxylabs/oxylabs.app.mjsOops! Something went wrong! :( ESLint: 8.57.1 Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (4)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (5)
components/oxylabs/actions/create-schedule/create-schedule.mjs (1)
12-12: Fix typo: "chron" should be "cron".The property name contains a typo that could cause confusion since it references cron expressions.
Apply this diff to fix the typo:
- chron: { + cron: {- cron: this.chron, + cron: this.cron,Also applies to: 36-36
components/oxylabs/sources/new-scheduled-run-completed/new-scheduled-run-completed.mjs (1)
28-34: Consider using run timestamp instead of current time.Using
Date.now()for the timestamp means all events will have nearly identical timestamps. If the run data includes an actual completion timestamp, that would be more meaningful for event ordering and debugging.generateMeta(run) { return { id: run.id, summary: `New Run with ID: ${run.id}`, - ts: Date.now(), + ts: run.completed_at ? new Date(run.completed_at).getTime() : Date.now(), }; },components/oxylabs/actions/create-proxy-session/create-proxy-session.mjs (1)
67-77: Refactor proxy URL construction for better readability.The current proxy URL construction is difficult to read and maintain. Consider breaking it down into smaller, more readable parts.
- const proxyUrl = `http://customer-${username}${cc - ? `-cc-${cc}` - : ""}${city - ? `-city-${city}` - : ""}${st - ? `-st-${st}` - : ""}${sessid - ? `-sessid-${sessid}` - : ""}${sstime - ? `-sstime-${sstime}` - : ""}:${password}@pr.oxylabs.io:7777`; + const urlParts = [`customer-${username}`]; + if (cc) urlParts.push(`cc-${cc}`); + if (city) urlParts.push(`city-${city}`); + if (st) urlParts.push(`st-${st}`); + if (sessid) urlParts.push(`sessid-${sessid}`); + if (sstime) urlParts.push(`sstime-${sstime}`); + + const proxyUrl = `http://${urlParts.join('-')}:${password}@pr.oxylabs.io:7777`;components/oxylabs/oxylabs.app.mjs (2)
20-20: Fix typo in description.There's a typo in the geoLocation description.
- description: "The geo locatio to scrape from. E.g. `United States`", + description: "The geo location to scrape from. E.g. `United States`",
56-65: Consider adding error handling for proxy connection failures.The createSession method creates an HTTPS proxy agent but doesn't handle potential proxy connection failures. This could result in unclear error messages for users.
async createSession({ $ = this, proxyUrl, ...opts }) { + try { const agent = new HttpsProxyAgent(proxyUrl); return axios($, { url: "https://ip.oxylabs.io/location", httpsAgent: agent, ...opts, }); + } catch (error) { + throw new Error(`Failed to create proxy session: ${error.message}`); + } },
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (12)
components/oxylabs/.gitignore(0 hunks)components/oxylabs/actions/create-proxy-session/create-proxy-session.mjs(1 hunks)components/oxylabs/actions/create-schedule/create-schedule.mjs(1 hunks)components/oxylabs/actions/scrape-url/scrape-url.mjs(1 hunks)components/oxylabs/actions/scrape-with-query/scrape-with-query.mjs(1 hunks)components/oxylabs/app/oxylabs.app.ts(0 hunks)components/oxylabs/common/constants.mjs(1 hunks)components/oxylabs/common/utils.mjs(1 hunks)components/oxylabs/oxylabs.app.mjs(1 hunks)components/oxylabs/package.json(1 hunks)components/oxylabs/sources/new-scheduled-run-completed/new-scheduled-run-completed.mjs(1 hunks)components/oxylabs/sources/new-scheduled-run-completed/test-event.mjs(1 hunks)
💤 Files with no reviewable changes (2)
- components/oxylabs/.gitignore
- components/oxylabs/app/oxylabs.app.ts
🧰 Additional context used
🪛 Biome (1.9.4)
components/oxylabs/sources/new-scheduled-run-completed/test-event.mjs
[error] 5-5: This number literal will lose precision at runtime.
The value at runtime will be 7300439540206948000
(lint/correctness/noPrecisionLoss)
[error] 12-12: This number literal will lose precision at runtime.
The value at runtime will be 7300439540169188000
(lint/correctness/noPrecisionLoss)
[error] 19-19: This number literal will lose precision at runtime.
The value at runtime will be 7300439540198552000
(lint/correctness/noPrecisionLoss)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: Verify TypeScript components
- GitHub Check: Publish TypeScript components
🔇 Additional comments (6)
components/oxylabs/common/utils.mjs (1)
1-25: LGTM! Clean recursive parsing utility.The
parseObjectfunction is well-implemented with proper error handling for JSON parsing and recursive processing of nested structures. The logic correctly handles all major data types.components/oxylabs/common/constants.mjs (1)
1-37: LGTM! Well-organized constants for Oxylabs sources.The constant definitions are comprehensive and well-structured. The separation between URL_SOURCES and QUERY_SOURCES provides clear categorization for different scraping use cases.
components/oxylabs/package.json (1)
3-3: LGTM! Package configuration properly updated for the migration.The version bump, main entry point change, and new dependencies appropriately reflect the migration from TypeScript to JavaScript and the addition of new functionality.
Also applies to: 5-5, 15-18
components/oxylabs/actions/create-schedule/create-schedule.mjs (1)
4-44: LGTM! Well-structured Pipedream action component.The action follows Pipedream conventions properly with clear property definitions, good documentation links, and appropriate use of the parseObject utility for input processing.
components/oxylabs/oxylabs.app.mjs (1)
12-15: ```shell
#!/bin/bashInspect the implementation of listSchedules in the Oxylabs component
rg -n "listSchedules" -C 5 components/oxylabs/oxylabs.app.mjs
</details> <details> <summary>components/oxylabs/actions/scrape-with-query/scrape-with-query.mjs (1)</summary> `4-42`: **Clean implementation following good patterns.** The action is well-structured with proper use of constants for source options, clear prop definitions, and appropriate API integration. The implementation follows Pipedream action conventions effectively. </details> </blockquote></details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
components/oxylabs/sources/new-scheduled-run-completed/new-scheduled-run-completed.mjs
Show resolved
Hide resolved
components/oxylabs/actions/create-proxy-session/create-proxy-session.mjs
Show resolved
Hide resolved
jcortes
left a comment
jcortes
left a comment
There was a problem hiding this comment.
@michelle0927 lgtm! Ready for QA!
|
/approve |
Resolves #5786
Actions Implemented
Sources Implemented
Skipped
Summary by CodeRabbit
Summary by CodeRabbit