Skip to content

Trawlingweb - new component #18887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 31, 2025
Merged

Trawlingweb - new component #18887

merged 2 commits into from
Oct 31, 2025

Conversation

@michelle0927
Copy link
Collaborator

@michelle0927 michelle0927 commented Oct 28, 2025 •
edited by coderabbitai bot
Loading

Resolves #10972

Summary by CodeRabbit

  • New Features

    • Added search news functionality to the Trawlingweb integration, allowing users to search news with customizable parameters including query, date range, sorting options, and result limits.

  • Chores

    • Updated component version and dependencies.

@vercel
Copy link

vercel bot commented Oct 28, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Preview Comments Updated (UTC)
pipedream-docs Ignored Ignored Oct 28, 2025 7:26pm
pipedream-docs-redirect-do-not-edit Ignored Ignored Oct 28, 2025 7:26pm

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 28, 2025 •
edited
Loading

Walkthrough

This PR adds a new Search News action to the Trawlingweb integration, refactors the API client to centralize HTTP request handling with automatic token injection, updates the package version and dependencies, effectively modularizing the integration layer.

Changes

Cohort / File(s) Summary
New Search News Action
components/trawlingweb/actions/search-news/search-news.mjs		 Exports a new action for searching news via Trawlingweb with configurable query parameters (dateStart, dateEnd, sort, order, maxResults), runs the search via the client, and returns results with a summary.
Refactored API Client
components/trawlingweb/trawlingweb.app.mjs		 Introduces _baseUrl() and _makeRequest() helper methods for centralized request handling with automatic token injection; delegates searchNews() to _makeRequest(); removes authKeys() method.
Package Configuration
components/trawlingweb/package.json		 Version bumped from 0.0.1 to 0.1.0; adds dependency on @pipedream/platform@^3.1.0.

Sequence Diagram

sequenceDiagram
    participant Action as Search News Action
    participant App as Trawlingweb App
    participant HTTP as HTTP Client (axios)
    participant API as Trawlingweb API

    Action->>Action: Assemble parameters<br/>(query, dateStart, dateEnd, etc.)
    Action->>App: searchNews(opts)
    activate App
    App->>App: _makeRequest(opts)
    activate App
    App->>App: Inject _baseUrl() & token
    App->>HTTP: axios.request({<br/>baseURL, params, token...})
    activate HTTP
    HTTP->>API: POST /search/news
    activate API
    API-->>HTTP: results[]
    deactivate API
    HTTP-->>App: response data
    deactivate HTTP
    deactivate App
    App-->>Action: data array
    deactivate App
    Action->>Action: Export summary
    Action-->>User: Return results
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

  • New action implementation: Verify parameter mapping, date parsing logic, and export structure align with Trawlingweb API expectations
  • Client refactoring: Ensure _makeRequest() properly injects authentication and that searchNews() delegation preserves existing behavior
  • Dependency integration: Confirm @pipedream/platform is the correct import source for axios and that version constraint is appropriate

Poem

🐰 Hop along the trawling way,
News retrieved throughout the day,
Requests bundled, tokens tight,
Search and sort with newfound might!
Version bumped, the flow takes flight! ✨

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings)
Check name Status Explanation Resolution
Description Check ⚠️ Warning The repository specifies a required template with a "WHY" section for pull request descriptions. The author-provided description only contains "Resolves #10972" and does not follow the template structure by filling in or explaining the "WHY" section. While the description does reference the linked issue, it fails to comply with the required template format and lacks substantive explanation of the purpose behind these changes.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Title Check ✅ Passed The title "Trawlingweb - new component" is concise, clear, and directly describes the primary change in the pull request. It accurately summarizes that a new Trawlingweb component is being added to the codebase. The title is specific and avoids vague terminology, making it immediately understandable to teammates reviewing the commit history.
Out of Scope Changes Check ✅ Passed All changes in the pull request directly support the implementation of the Trawlingweb component. The modifications include the new search-news action file, updates to the Trawlingweb app module to support the action, and necessary package.json updates for dependencies and versioning. No unrelated or out-of-scope changes are present in the changeset.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch issue-10972
📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 36a8917 and 859cc8b.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (3)
  • components/trawlingweb/actions/search-news/search-news.mjs (1 hunks)
  • components/trawlingweb/package.json (2 hunks)
  • components/trawlingweb/trawlingweb.app.mjs (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
  • GitHub Check: Publish TypeScript components
  • GitHub Check: Verify TypeScript components
  • GitHub Check: pnpm publish
  • GitHub Check: Lint Code Base
🔇 Additional comments (8)
components/trawlingweb/package.json (1)

3-3: LGTM!

The version bump to 0.1.0 is appropriate for the new action functionality, and the @pipedream/platform dependency is correctly added to support the axios usage in the app module.

Also applies to: 14-17

components/trawlingweb/trawlingweb.app.mjs (3)

1-1: LGTM!

The axios import from @pipedream/platform is correctly added to support the HTTP request functionality.

23-27: LGTM!

The delegation pattern is clean and allows flexibility by passing through all options to the underlying request method.

8-22: The code implementation is correct for the Trawlingweb API.

The Trawlingweb API expects requests to the base URL https://api.trawlingweb.com with search terms passed as query parameters (token for API key and q for the query). The code correctly constructs this by setting the url to the base URL and passing token and other parameters through the params object, which axios will convert to query parameters in the request.

Likely an incorrect or invalid review comment.

components/trawlingweb/actions/search-news/search-news.mjs (4)

1-13: LGTM!

The action metadata and structure follow Pipedream conventions correctly, with appropriate annotations for a read-only search operation.

14-60: LGTM!

The props are well-defined with clear labels, descriptions, and appropriate type constraints. The max: 100 validation on maxResults is a good safeguard.

62-62: Verify API response structure.

The code assumes the axios response has the shape { response: { data } }. Please confirm this matches the actual Trawlingweb API response structure. The optional chaining on line 80 (data?.length) is good defensive coding.

Also applies to: 80-82

66-74: Verify date format and sort parameter logic against official API documentation.

The code contains a discrepancy that requires verification:

  1. Date format conflict: Props describe dateStart and dateEnd as "ISO 8601 format (YYYY-MM-DD)", but the code converts these strings to millisecond timestamps using Date.parse(). Verify against the official Trawlingweb API documentation whether the ts and tsi parameters expect ISO date strings or millisecond timestamps.

  2. Sort parameter ambiguity: The code sets sort: "published" only when selected, otherwise undefined. Confirm whether sending undefined (or omitting the parameter) for the "crawled" sort option is the intended API behavior, or if the code should pass this.sort directly.

Without access to the official API documentation, these behaviors cannot be confirmed as correct. Please verify the expected parameter formats in the Trawlingweb dashboard and update the code accordingly.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

GTFalcao
GTFalcao approved these changes Oct 30, 2025
View reviewed changes
Copy link
Collaborator

@GTFalcao GTFalcao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@vunguyenhung vunguyenhung merged commit 3d98934 into master Oct 31, 2025
10 checks passed
@vunguyenhung vunguyenhung deleted the issue-10972 branch October 31, 2025 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@GTFalcao GTFalcao GTFalcao approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Components] trawlingweb

4 participants

@michelle0927 @GTFalcao @vunguyenhung