feat(source-Monday): Update CDK to v6 by tolik0 · Pull Request #55225 · airbytehq/airbyte

Anatolii Yatsuk (tolik0) · 2025-03-06T12:30:49Z

What

Update CDK to v6.

How

Migrated boards and items stream to StateDelegatingStream.
Update custom components to match abstract classes from the CDK

User Impact

The sync should become quicker as the source is now concurrent.

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

vercel · 2025-03-06T12:30:54Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 24, 2025 1:17pm

Natik Gadzhi (natikgadzhi) · 2025-03-09T19:31:09Z

Anatolii Yatsuk (@tolik0) is this ready to go? If so, David Gold (@dbgold17) can you review this and see if we should merge this first and apply your changes on top, so you get to work with the new CDK?

David Gold (dbgold17)

Hi Anatolii Yatsuk (@tolik0) , I left a bunch of comments, a lot of small things and then some which have to do with me not understanding the changes you're making. Would you be able to write a couple sentences on the PR description about the structural changes you needed to make in order to update the CDK to version 6? It would be super helpful for me and for others looking at this code in the future :)

Also, how have you tested these changes? Do you feel confident that they won't break anything for uses?

airbyte-integrations/connectors/source-monday/source_monday/source.py

airbyte-integrations/connectors/source-monday/source_monday/run.py

David Gold (dbgold17) · 2025-03-10T05:38:16Z

airbyte-integrations/connectors/source-monday/pyproject.toml

-python = "^3.9,<3.12"
-airbyte-cdk = "0.78.6" # Breaks with newer versions of the CDK
+python = ">=3.10,<3.12"
+airbyte_cdk = { path = "/home/anatolii/airbytehq/airbyte/airbyte-cdk/python/airbyte-python-cdk/", develop = true }


this should be replaced with a public package version right?

Added dev version

David Gold (dbgold17) · 2025-03-10T05:39:28Z

airbyte-integrations/connectors/source-monday/metadata.yaml

  connectorType: source
  definitionId: 80a54ea2-9959-4040-aac1-eee42423ec9b
-  dockerImageTag: 2.1.13
+  dockerImageTag: 2.2.0


is there consensus about whether to bump minor or patch number in this version?

This should be the minor update

David Gold (dbgold17) · 2025-03-10T05:43:29Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

-            return extracted if isinstance(extracted, list) else [extracted]
-        return []
+        decoded_response = self.decoder.decode(response)
+        for response_body in decoded_response:


why are you now looping over this response where we weren't before?

The underlying CDK class changed

airbyte-integrations/connectors/source-monday/source_monday/graphql_requester.py

David Gold (dbgold17) · 2025-03-10T05:58:34Z

airbyte-integrations/connectors/source-monday/source_monday/item_pagination_strategy.py

+        last_page_size: int,
+        last_record: Optional[Record],
+        last_page_token_value: Optional[Any]) -> Optional[Tuple[Optional[int], Optional[int]]]:
        """


update docstring to conform with new interface.

David Gold (dbgold17) · 2025-03-10T05:58:35Z

airbyte-integrations/connectors/source-monday/source_monday/item_pagination_strategy.py

        self._sub_page: Optional[int] = self.start_from_page

-    def next_page_token(self, response, last_records: List[Mapping[str, Any]]) -> Optional[Tuple[Optional[int], Optional[int]]]:
+    def next_page_token(self,


why is this change necessary? It doesn't seem like you're using any of these arguments.

If it's only in order to conform to some interface spec, can you note that in a comment?

Yes, the underlying class in the CDK changed, so we need to align with it

David Gold (dbgold17) · 2025-03-10T06:15:32Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

+        response_decoded = self.decoder.decode(response)
+        for response_body in response_decoded:
+            if not response_body["data"]["boards"]:
+                yield from []


do you actually want to return a generator object with nothing in it here or just continue?

David Gold (dbgold17) · 2025-03-10T06:17:12Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

@@ -37,32 +37,30 @@ class MondayActivityExtractor(RecordExtractor):
    decoder: Decoder = JsonDecoder(parameters={})

    def extract_records(self, response: requests.Response) -> List[Record]:


should the return type annotation be updated?

Anatolii Yatsuk (tolik0) · 2025-03-10T09:58:14Z

David Gold (@dbgold17) Sorry, this PR is not in review yet. I still need to add StateDelegatingRetriever, which hasn’t been merged yet.
I'm currently busy handling P1 issues, but I’ll ping you once it’s ready.
Thank you!

David Gold (dbgold17) · 2025-03-10T17:11:54Z

David Gold (@dbgold17) Sorry, this PR is not in review yet. I still need to add StateDelegatingRetriever, which hasn’t been merged yet. I'm currently busy handling P1 issues, but I’ll ping you once it’s ready. Thank you!

Sounds good, sorry for the premature review!

…n-router

Anatolii Yatsuk (tolik0) · 2025-03-18T14:51:04Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (1264ab2)

Anatolii Yatsuk (tolik0) · 2025-03-20T10:25:07Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (9d4234d)

Anatolii Yatsuk (tolik0) · 2025-03-20T18:31:22Z

Regression test results:

… into tolik0/source-monday/add-grouping-partition-router

Brian Lai (brianjlai)

one suggestions on refactoring try_extract_records() to process records iteratively since the underlying call became a generator, but otherwise no other comments and looks good to me!

Brian Lai (brianjlai) · 2025-03-20T19:03:41Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

-                return [x for x in extracted if x]
-            return extracted if isinstance(extracted, list) else [extracted]
-        return []
+        decoded_response = self.decoder.decode(response)


because self.decoder.decode(response) has been changed to become a generator, can we refactor this method to also iterate over the generator and yield from extracted below instead of storing them in the result list?

Brian Lai (brianjlai) · 2025-03-20T19:08:36Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

                    values["text"] = display_value

-        return result
+        yield from result


In addition to the above comment, if we refactor self.try_extract_records() to be an iterable, we should also yield record at the end of each iteration of the for loop so we don't have to keep everything in memory with result

Brian Lai (brianjlai) · 2025-03-20T19:13:40Z

airbyte-integrations/connectors/source-monday/source_monday/manifest.yaml

+concurrency_level:
+  type: ConcurrencyLevel
+  default_concurrency: "{{ config.get('num_workers', 4) }}"
+  max_concurrency: 40


Monday actually has a pretty detailed section on rate limits: https://developer.monday.com/api-reference/docs/rate-limits

40 actually seems like an acceptable threshold since they call out only allowing for at most 40 concurrent requests at any given moment. But let's directly reference https://developer.monday.com/api-reference/docs/rate-limits#concurrency-limit this in the comment

Anatolii Yatsuk (tolik0) · 2025-03-21T13:24:41Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (6adeeb1)

Brian Lai (brianjlai)

i had a couple small questions, but i'm fine with just answering or addressing and no need to block.

Brian Lai (brianjlai) · 2025-03-21T17:38:07Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

+            else:
+                path = [p.eval(self.config) for p in field_path]
+                if "*" in path:
+                    extracted = dpath.values(body, path)


small nit/question, why the switch to use dpath.values() instead of dpath.util.values()? same for the below get() as well

Updated according to the latest implementation of DpathExtractor. Here is the PR for the util change: #38847

Brian Lai (brianjlai) · 2025-03-21T17:48:56Z

airbyte-integrations/connectors/source-monday/source_monday/extractor.py

-            result = self.try_extract_records(response, self.field_path_incremental)
-
-        for record in result:
-            if "updated_at" in record:


why don't we have to edit some of the fields on each record object anymore?

I moved this to transformations

ah got it, must've missed that. thanks for clarifying

Anatolii Yatsuk (tolik0) · 2025-03-24T13:09:37Z

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (8fbc8f7)

Anatolii Yatsuk (tolik0) added 7 commits November 21, 2024 15:29

Add WaitTimeFromHeader backoff strategy

9b58f82

Update version

c11239c

Update CDK version to 0.90.0

2382b81

Revert CDK update

163d3dc

Update page size for items stream

1a3c390

Update page size to 10

7de9fa8

Update CDK to v6

c9cc1b5

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/monday labels Mar 6, 2025

vercel bot deployed to Preview March 6, 2025 12:34 View deployment

David Gold (dbgold17) reviewed Mar 10, 2025

View reviewed changes

Anatolii Yatsuk (tolik0) marked this pull request as draft March 10, 2025 09:55

David Gold (dbgold17) mentioned this pull request Mar 12, 2025

[WIP] upgrade source monday to cdk v6 #55712

Closed

2 tasks

Anatolii Yatsuk (tolik0) added 2 commits March 18, 2025 16:32

Add StateDelegatingStream

4434e5d

Merge branch 'master' into tolik0/source-monday/add-grouping-partitio…

633bcaa

…n-router

vercel bot deployed to Preview March 18, 2025 14:39 View deployment

Anatolii Yatsuk (tolik0) added 2 commits March 18, 2025 16:45

Update Monday version

7fde793

Update spec

c2036a2

Octavia Squidington III (octavia-squidington-iii) and others added 2 commits March 18, 2025 14:52

chore: auto-fix lint and format issues

1264ab2

Update poetry.lock

ef34439

vercel bot deployed to Preview March 18, 2025 15:11 View deployment

Anatolii Yatsuk (tolik0) added 2 commits March 18, 2025 17:19

Small refactoring

f9f53cc

Fix types in custom record extractor

fd46351

Anatolii Yatsuk (tolik0) marked this pull request as ready for review March 18, 2025 15:25

vercel bot deployed to Preview March 19, 2025 17:43 View deployment

Fix state migration for boards stream

fbf1503

vercel bot deployed to Preview March 19, 2025 18:11 View deployment

Fix unit test coverage

fbe8c79

chore: auto-fix lint and format issues

9d4234d

vercel bot deployed to Preview March 20, 2025 10:37 View deployment

Fix state migration test

5c7cc55

vercel bot deployed to Preview March 20, 2025 11:27 View deployment

Merge branch 'tolik0/source-monday/add-wait-time-from-header-backoff'…

23c957a

… into tolik0/source-monday/add-grouping-partition-router

Anatolii Yatsuk (tolik0) requested a review from Brian Lai (brianjlai) March 20, 2025 18:36

Update CDK version

62fce0b

vercel bot deployed to Preview March 20, 2025 18:50 View deployment

Brian Lai (brianjlai) reviewed Mar 20, 2025

View reviewed changes

Refactor MondayIncrementalItemsExtractor

edccab3

chore: auto-fix lint and format issues

6adeeb1

vercel bot deployed to Preview March 21, 2025 13:35 View deployment

Anatolii Yatsuk (tolik0) requested a review from Brian Lai (brianjlai) March 21, 2025 15:23

Brian Lai (brianjlai) approved these changes Mar 21, 2025

View reviewed changes

vercel bot deployed to Preview March 24, 2025 12:42 View deployment

Enable progressive rollout

ef8b7e3

chore: auto-fix lint and format issues

8fbc8f7

vercel bot deployed to Preview March 24, 2025 13:17 View deployment

Anatolii Yatsuk (tolik0) merged commit f1b6f66 into master Mar 24, 2025
27 checks passed

Anatolii Yatsuk (tolik0) deleted the tolik0/source-monday/add-grouping-partition-router branch March 24, 2025 16:18

Sven Pöche (Valgard) pushed a commit to mayflower/airbyte that referenced this pull request Mar 25, 2025

feat(source-Monday): Update CDK to v6 (airbytehq#55225)

b3b5bd4

Maxime Carbonneau-Leclerc (maxi297) mentioned this pull request Mar 28, 2025

fix(source-monday): Add WaitTimeFromHeader backoff strategy #48597

Closed

2 tasks

		@@ -37,32 +37,30 @@ class MondayActivityExtractor(RecordExtractor):
		decoder: Decoder = JsonDecoder(parameters={})

		def extract_records(self, response: requests.Response) -> List[Record]:

Conversation

Anatolii Yatsuk (tolik0) commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

User Impact

Can this PR be safely reverted and rolled back?

Uh oh!

vercel bot commented Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Natik Gadzhi (natikgadzhi) commented Mar 9, 2025

Uh oh!

David Gold (dbgold17) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

David Gold (dbgold17) Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anatolii Yatsuk (tolik0) commented Mar 10, 2025

Uh oh!

David Gold (dbgold17) commented Mar 10, 2025

Uh oh!

Anatolii Yatsuk (tolik0) commented Mar 18, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anatolii Yatsuk (tolik0) commented Mar 20, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Anatolii Yatsuk (tolik0) commented Mar 20, 2025

Uh oh!

Brian Lai (brianjlai) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Brian Lai (brianjlai) Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anatolii Yatsuk (tolik0) commented Mar 21, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Brian Lai (brianjlai) left a comment

Choose a reason for hiding this comment

Uh oh!

Anatolii Yatsuk (tolik0) commented Mar 6, 2025 •

edited

Loading

vercel bot commented Mar 6, 2025 •

edited

Loading

David Gold (dbgold17) Mar 10, 2025 •

edited

Loading

Anatolii Yatsuk (tolik0) commented Mar 18, 2025 •

edited by github-actions bot

Loading

Anatolii Yatsuk (tolik0) commented Mar 20, 2025 •

edited by github-actions bot

Loading

Brian Lai (brianjlai) Mar 20, 2025 •

edited

Loading

Anatolii Yatsuk (tolik0) commented Mar 21, 2025 •

edited by github-actions bot

Loading

Anatolii Yatsuk (tolik0) commented Mar 24, 2025 •

edited by github-actions bot

Loading