Skip to content

Commit 4b9783e

Browse files
chore(CE): Allow Batch Support and Batch Size for S3 (#941)
* Resolve conflict in cherry-pick of cb46584b60b87b9408cb5dcdc84133d956aa844d and change the commit message * chore(CE): Resolved conflict --------- Co-authored-by: TivonB-AI2 <124182151+TivonB-AI2@users.noreply.github.com> Co-authored-by: TivonB-AI2 <tivon.brown@squared.ai>
1 parent 5ce7fed commit 4b9783e

File tree

4 files changed

+9
-7
lines changed

4 files changed

+9
-7
lines changed

integrations/Gemfile.lock

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ GIT
77
PATH
88
remote: .
99
specs:
10-
multiwoven-integrations (0.34.17)
10+
multiwoven-integrations (0.34.18)
1111
MailchimpMarketing
1212
activesupport
1313
async-websocket

integrations/lib/multiwoven/integrations/destination/amazon_s3/client.rb

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,9 @@ def create_streams(tables)
154154
Multiwoven::Integrations::Protocol::Stream.new(
155155
name: r[:tablename],
156156
action: StreamAction["create"],
157-
json_schema: convert_to_json_schema(r[:columns])
157+
json_schema: convert_to_json_schema(r[:columns]),
158+
batch_support: true,
159+
batch_size: 100_000
158160
)
159161
end
160162
end

integrations/lib/multiwoven/integrations/rollout.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
module Multiwoven
44
module Integrations
5-
VERSION = "0.34.17"
5+
VERSION = "0.34.18"
66

77
ENABLED_SOURCES = %w[
88
Snowflake

integrations/spec/multiwoven/integrations/destination/amazon_s3/client_spec.rb

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,8 @@
135135
expect(catalog.streams.count).to eql(1)
136136
expect(catalog.schema_mode).to eql("schema")
137137
expect(catalog.streams[0].name).to eql("test_file")
138-
expect(catalog.streams[0].batch_support).to eql(false)
139-
expect(catalog.streams[0].batch_size).to eql(1)
138+
expect(catalog.streams[0].batch_support).to eql(true)
139+
expect(catalog.streams[0].batch_size).to eql(100_000)
140140
expect(catalog.streams[0].supported_sync_modes).to eql(%w[incremental])
141141
end
142142

@@ -176,8 +176,8 @@
176176
expect(message.catalog.streams.count).to eql(1)
177177
expect(message.catalog.schema_mode).to eql("schema")
178178
expect(message.catalog.streams[0].name).to eql("test_file")
179-
expect(message.catalog.streams[0].batch_support).to eql(false)
180-
expect(message.catalog.streams[0].batch_size).to eql(1)
179+
expect(message.catalog.streams[0].batch_support).to eql(true)
180+
expect(message.catalog.streams[0].batch_size).to eql(100_000)
181181
expect(message.catalog.streams[0].supported_sync_modes).to eql(%w[incremental])
182182
end
183183
end

0 commit comments

Comments
 (0)