Skip to content

Conversation

@luancazarine
Copy link
Collaborator

@luancazarine luancazarine commented Feb 11, 2025

Resolves #15141.

Summary by CodeRabbit

  • New Features
    • Introduced capabilities to retrieve company financial details, historical price data, and current stock prices using stock ticker symbols.
    • Enhanced market data integration with improved pagination, dynamic ticker listings, and configurable filtering and sorting options.
    • Added new modules for handling market news, stock price summaries, and trade events for better insights.
    • Implemented new properties and methods for improved data interaction and retrieval within the polygon component.
    • Added constants for standardized time spans, timeframes, and sorting options.

@luancazarine luancazarine added the ai-assisted Content generated by AI, with human refinement and modification label Feb 11, 2025
@vercel
Copy link

vercel bot commented Feb 11, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

3 Skipped Deployments
Name Status Preview Comments Updated (UTC)
docs-v2 ⬜️ Ignored (Inspect) Visit Preview Feb 20, 2025 7:02pm
pipedream-docs ⬜️ Ignored (Inspect) Feb 20, 2025 7:02pm
pipedream-docs-redirect-do-not-edit ⬜️ Ignored (Inspect) Feb 20, 2025 7:02pm

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 11, 2025

Walkthrough

Multiple new modules and enhancements have been added across the Polygon component. New action modules enable retrieval of company financials, historical prices, and stock prices via the Polygon API. Common utilities and constants have been introduced, and the main app component now includes additional properties and methods for API requests and pagination. Furthermore, new sources for market news, stock price summaries, and stock trade events—along with corresponding test events—have been implemented. A new package manifest now defines the component’s metadata and dependencies.

Changes

File(s) Change Summary
components/polygon/actions/…/get-company-financials.mjs, …/get-historical-prices.mjs, …/get-stock-price.mjs Introduce new action modules to fetch company financials, historical prices, and stock prices via the Polygon API with defined metadata, props, and error handling.
components/polygon/common/…/constants.mjs, …/utils.mjs Add new constants (time spans, timeframes, sort options) and introduce the parseNextPage utility function for URL pagination.
components/polygon/polygon.app.mjs Enhance the Polygon app with new properties (stockTicker, adjusted) and methods (_baseUrl, _headers, _makeRequest, various getters, and paginate) for improved API request management and data retrieval.
components/polygon/package.json Create a new package manifest with component metadata, dependency (@pipedream/platform), and publish configuration.
components/polygon/sources/common/base.mjs Introduce a base module to handle event emissions, database state management (last processed date), and paginated data retrieval.
components/polygon/sources/new-market-news/… Add a new source module for fetching and processing market news events with pagination and include a test event file.
components/polygon/sources/new-stock-price-summary/… Implement a new source module for summarizing stock price data with methods for fetching, summarizing, and parsing; includes test event data.
components/polygon/sources/new-stock-trade/… Implement a new source module for processing stock trade events with methods for retrieving and summarizing trade data; includes test event file.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant App as Polygon App
    participant API as Polygon API
    User->>App: Call getCurrentPrice(stockTicker, date)
    App->>API: _makeRequest(opts)
    API-->>App: Return stock price data
    App-->>User: Return data with summary message
Loading
sequenceDiagram
    participant Scheduler
    participant Base as Base Module
    participant DB as Database
    participant API as Data API
    Scheduler->>Base: Trigger emitEvent/run method
    Base->>DB: _getLastDate()
    Base->>API: Fetch paginated data (paginate)
    API-->>Base: Return results with next page info
    Base->>DB: _setLastDate() if new items found
    Base-->>Scheduler: Emit event for each new item
Loading

Suggested reviewers

  • michelle0927

Poem

Hi, I'm a little rabbit, hopping with delight,
In a field of code changes, everything's just right.
New modules sprout like carrots in the ground,
With API flows and pagination all around.
I twirl my ears in joy for updates so bright!
🐰✨

Warning

There were issues while running some tools. Please review the errors and either fix the tool’s configuration or disable the tool if it’s a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

components/bitbucket_data_center/bitbucket_data_center.app.mjs

Oops! Something went wrong! :(

ESLint: 8.57.1

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs
at packageResolve (node:internal/modules/esm/resolve:839:9)
at moduleResolve (node:internal/modules/esm/resolve:908:18)
at defaultResolve (node:internal/modules/esm/resolve:1038:11)
at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:557:12)
at ModuleLoader.resolve (node:internal/modules/esm/loader:525:25)
at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:246:38)
at ModuleJob._link (node:internal/modules/esm/module_job:126:49)

components/brillium/brillium.app.mjs

Oops! Something went wrong! :(

ESLint: 8.57.1

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'jsonc-eslint-parser' imported from /eslint.config.mjs
at packageResolve (node:internal/modules/esm/resolve:839:9)
at moduleResolve (node:internal/modules/esm/resolve:908:18)
at defaultResolve (node:internal/modules/esm/resolve:1038:11)
at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:557:12)
at ModuleLoader.resolve (node:internal/modules/esm/loader:525:25)
at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:246:38)
at ModuleJob._link (node:internal/modules/esm/module_job:126:49)


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 19f4866 and 838fdde.

📒 Files selected for processing (2)
  • components/bitbucket_data_center/bitbucket_data_center.app.mjs (1 hunks)
  • components/brillium/brillium.app.mjs (1 hunks)
✅ Files skipped from review due to trivial changes (2)
  • components/bitbucket_data_center/bitbucket_data_center.app.mjs
  • components/brillium/brillium.app.mjs
⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: Publish TypeScript components
  • GitHub Check: pnpm publish
  • GitHub Check: Verify TypeScript components

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Sources
 - New Market Trade
 - New Market News
 - New Stock Price Summary

Actions
 - Get Stock Price
 - Get Historical Prices
 - Get Company
@luancazarine luancazarine marked this pull request as ready for review February 13, 2025 16:38
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🧹 Nitpick comments (12)
components/polygon/polygon.app.mjs (3)

7-44: Consider adding a default value for adjusted.
Since "By default, results are adjusted" per the description, you may wish to explicitly set default: true for the adjusted prop to prevent future ambiguity and ensure consistent behavior if the user doesn't select a value.


46-62: Add explicit error handling to _makeRequest.
Returning raw responses from APIs can obscure root causes when requests fail. Consider wrapping your axios call in a try/catch block to handle and log errors more gracefully.


113-141: Prevent potential infinite loop in pagination.
Consider implementing an iteration boundary or a timeout if parseDataFn or the API incorrectly returns a nextPage for all pages. This ensures the loop won't run indefinitely in the event of unexpected responses.

components/polygon/common/utils.mjs (1)

1-7: Handle invalid URL scenarios gracefully.
If nextUrl is sometimes malformed, calling new URL(nextUrl) may throw an error. Checking for valid URLs beforehand or wrapping it in a try/catch could help prevent unexpected runtime failures.

components/polygon/sources/new-market-news/new-market-news.mjs (1)

24-27: Consider adjusting the limit parameter.

Setting a high limit of 1000 items per request might impact performance and memory usage.

Consider reducing the limit and relying on pagination:

-          "limit": 1000,
+          "limit": 100,
components/polygon/sources/new-stock-trade/new-stock-trade.mjs (1)

33-35: Add price formatting.

The price in the summary should be formatted for better readability.

-      return `New trade for ${item.sym} at $${item.p}`;
+      return `New trade for ${item.sym} at $${item.p.toFixed(2)}`;
components/polygon/sources/new-stock-price-summary/new-stock-price-summary.mjs (2)

34-35: Consider adjusting the limit parameter.

Setting a high limit of 5000 items per request might impact performance and memory usage.

Consider reducing the limit and relying on pagination:

-          "limit": 5000,
+          "limit": 100,

27-29: Enhance summary with price information.

The summary could be more informative by including the actual price data.

-    getSummary() {
-      return `Daily price summary for ${this.stockTicker}`;
+    getSummary(item) {
+      return `Daily price summary for ${this.stockTicker}: Open $${item.o.toFixed(2)}, Close $${item.c.toFixed(2)}`;
     },
components/polygon/sources/common/base.mjs (2)

37-37: Use a more reliable date comparison method.

Date.parse() can be unreliable with certain date formats. Consider using a date parsing library like date-fns or moment.js for more robust date handling.


46-46: Consider timezone handling in timestamp fallback.

The fallback to new Date() doesn't consider timezone differences. Consider using UTC dates consistently.

-          const ts = Date.parse(item.published_utc || item.sip_timestamp || new Date());
+          const ts = Date.parse(item.published_utc || item.sip_timestamp || new Date().toISOString());
components/polygon/actions/get-company-financials/get-company-financials.mjs (2)

24-28: Improve description formatting.

The filing date description uses template literals with manual line breaks. Consider using a more maintainable format.

-      description: `Query by the date when the filing with financials data was filed in **YYYY-MM-DD** format.
-                  \nBest used when querying over date ranges to find financials based on filings that happen in a time period.
-                  \nExamples:
-                  \nTo get financials based on filings that have happened after January 1, 2009 use the query param filing_date.gte=2009-01-01
-                  \nTo get financials based on filings that happened in the year 2009 use the query params filing_date.gte=2009-01-01&filing_date.lt=2010-01-01`,
+      description: [
+        "Query by the date when the filing with financials data was filed in **YYYY-MM-DD** format.",
+        "Best used when querying over date ranges to find financials based on filings that happen in a time period.",
+        "Examples:",
+        "- To get financials based on filings that have happened after January 1, 2009 use the query param filing_date.gte=2009-01-01",
+        "- To get financials based on filings that happened in the year 2009 use the query params filing_date.gte=2009-01-01&filing_date.lt=2010-01-01"
+      ].join("\n"),

71-79: Simplify params object structure.

The params object includes the stockTicker which could be passed directly to the API call.

     const financialDetails = await this.polygon.getFinancialDetails({
       $,
+      stockTicker: this.stockTicker,
       params: {
-        stockTicker: this.stockTicker,
         filingDate: this.filingDate,
         periodOfReportDate: this.periodOfReportDate,
         timeframe: this.timeframe,
         order: this.order,
         limit: this.limit,
         sort: this.sort,
       },
     });
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e51b8b1 and db53c01.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (14)
  • components/polygon/actions/get-company-financials/get-company-financials.mjs (1 hunks)
  • components/polygon/actions/get-historical-prices/get-historical-prices.mjs (1 hunks)
  • components/polygon/actions/get-stock-price/get-stock-price.mjs (1 hunks)
  • components/polygon/common/constants.mjs (1 hunks)
  • components/polygon/common/utils.mjs (1 hunks)
  • components/polygon/package.json (1 hunks)
  • components/polygon/polygon.app.mjs (1 hunks)
  • components/polygon/sources/common/base.mjs (1 hunks)
  • components/polygon/sources/new-market-news/new-market-news.mjs (1 hunks)
  • components/polygon/sources/new-market-news/test-event.mjs (1 hunks)
  • components/polygon/sources/new-stock-price-summary/new-stock-price-summary.mjs (1 hunks)
  • components/polygon/sources/new-stock-price-summary/test-event.mjs (1 hunks)
  • components/polygon/sources/new-stock-trade/new-stock-trade.mjs (1 hunks)
  • components/polygon/sources/new-stock-trade/test-event.mjs (1 hunks)
✅ Files skipped from review due to trivial changes (3)
  • components/polygon/package.json
  • components/polygon/common/constants.mjs
  • components/polygon/sources/new-stock-price-summary/test-event.mjs
🔇 Additional comments (9)
components/polygon/polygon.app.mjs (2)

1-2: Imports appear correct.
These new imports from "@pipedream/platform" and the local utility module look consistent with the rest of the codebase.


63-112: Verify endpoint correctness and use of adjusted parameter.
Multiple methods (e.g., getCurrentPrice, getHistoricalPriceData, etc.) do not appear to pass adjusted in the request params. If the API supports an adjusted option, confirm that omitting it is intentional.

components/polygon/sources/new-stock-trade/test-event.mjs (1)

1-14: Data structure for the test event looks good.
Providing realistic trade fields can help validate the expected shape of stock trade data. This aligns with typical usage in source tests.

components/polygon/sources/new-market-news/new-market-news.mjs (2)

1-12: LGTM! Well-structured module definition.

The module is well-organized with clear imports and proper metadata configuration. The description accurately reflects the module's purpose.


31-40: Consider handling empty results.

The parseData method should handle cases where results array is empty or undefined.

Add validation to ensure robust error handling:

 parseData({
   results, next_url: next,
 }) {
+  if (!results || !Array.isArray(results)) {
+    return {
+      parsedData: [],
+      nextPage: null,
+    };
+  }
   const parsedPage = parseNextPage(next);

   return {
     parsedData: results,
     nextPage: parsedPage,
   };
 }
components/polygon/sources/new-stock-trade/new-stock-trade.mjs (2)

30-32: Verify method name consistency.

The method name getPreviousClose seems inconsistent with the module's purpose of emitting new trade events.

Please verify if this is the correct method to use for fetching new trade data.


42-47: Consider validating results.

Similar to the market news source, add validation for the results array.

 parseData({ results }) {
+  if (!results || !Array.isArray(results)) {
+    return {
+      parsedData: [],
+      nextPage: null,
+    };
+  }
   return {
     parsedData: results,
     nextPage: null,
   };
 }
components/polygon/sources/new-stock-price-summary/new-stock-price-summary.mjs (1)

41-50: Consider validating results.

Similar to other sources, add validation for the results array.

 parseData({
   results, next_url: next,
 }) {
+  if (!results || !Array.isArray(results)) {
+    return {
+      parsedData: [],
+      nextPage: null,
+    };
+  }
   const parsedPage = parseNextPage(next);

   return {
     parsedData: results,
     nextPage: parsedPage,
   };
 }
components/polygon/sources/new-market-news/test-event.mjs (1)

20-20: Update the test event date to a past date.

The test event uses a future date (2024-06-24T18:33:53Z). This could cause issues in tests that compare dates or check for recent events. Consider using a past date to ensure consistent test behavior.

…l-prices.mjs

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
michelle0927
michelle0927 previously approved these changes Feb 18, 2025
Copy link
Collaborator

@michelle0927 michelle0927 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@luancazarine
Copy link
Collaborator Author

/approve

@luancazarine luancazarine merged commit 787abff into master Feb 20, 2025
11 checks passed
@luancazarine luancazarine deleted the issue-15141 branch February 20, 2025 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted Content generated by AI, with human refinement and modification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Components] polygon

3 participants