Skip to content

Comments

chore: Double the limit of grpc messages#2131

Merged
przste-go merged 5 commits intomainfrom
transformer-limits
Apr 28, 2025
Merged

chore: Double the limit of grpc messages#2131
przste-go merged 5 commits intomainfrom
transformer-limits

Conversation

@przste-go
Copy link

Summary

Following up on the transformer hanging issue reported by a customer and addressed in PR #2130:
Currently, the transformer plugin can increase the message size beyond the limit enforced in the sync method of plugin-sdk 1.
To accommodate scenarios where transformers or destination plugins need to handle larger messages than the source plugin, we should consider increasing this limit to provide more flexibility.


Use the following steps to ensure your PR is ready to be reviewed

  • Read the contribution guidelines 🧑‍🎓
  • Run go fmt to format your code 🖊
  • Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
  • Update or add tests 🧪
  • Ensure the status checks below are successful ✅

@przste-go przste-go requested review from a team and blesniewski April 28, 2025 08:36
@murarustefaan
Copy link
Member

murarustefaan commented Apr 28, 2025

I'm wondering if we need to change it here as well; probably not required, since serve wraps around the plugin either way.
Worth checking, we could change it for consistency reasons.
https://github.com/cloudquery/plugin-pb-go/blob/main/managedplugin/plugin.go#L39

Co-authored-by: Bartosz Lesniewski <bartosz@cloudquery.io>
@przste-go
Copy link
Author

I'm wondering if we need to change it here as well; probably not required, since serve wraps around the plugin either way. Worth checking, we could change it for consistency reasons. https://github.com/cloudquery/plugin-pb-go/blob/main/managedplugin/plugin.go#L39

@murarustefaan Not sure here, I have to find out how that code is actually used.

@przste-go
Copy link
Author

@murarustefaan I think you are right, this could be also needed due to destination client size limit in CLI.
I opened another PR
cloudquery/plugin-pb-go#494

@murarustefaan
Copy link
Member

Nice! Then let's merge that one, get it out and upgrade the dependency version in the SDK here and only then get this one merged.

@przste-go
Copy link
Author

@murarustefaan Released and updated

@przste-go
Copy link
Author

Seems like I have to update plugins first 🤔

@przste-go przste-go merged commit 12beabc into main Apr 28, 2025
11 checks passed
@przste-go przste-go deleted the transformer-limits branch April 28, 2025 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants