Skip to content

Commit 26c19e7

Browse files
aisquared-ci-bot[bot]RafaelOAiSquaredTivonB-AI2
authored
feat(CE): refactor google drive source connector (#895)
* Resolve conflict in cherry-pick of a21a688041c356ae5425e1af6787915778da9b82 and change the commit message * chore(CE): Resolved conflict --------- Co-authored-by: Rafael E. O'Neill <106079170+RafaelOAiSquared@users.noreply.github.com> Co-authored-by: TivonB-AI2 <tivon.brown@squared.ai>
1 parent 1fecf67 commit 26c19e7

File tree

7 files changed

+375
-505
lines changed

7 files changed

+375
-505
lines changed

integrations/Gemfile.lock

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ GIT
77
PATH
88
remote: .
99
specs:
10-
multiwoven-integrations (0.34.8)
10+
multiwoven-integrations (0.34.9)
1111
MailchimpMarketing
1212
activesupport
1313
async-websocket

integrations/lib/multiwoven/integrations/core/unstructured_source_connector.rb

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,15 +30,22 @@ class UnstructuredSourceConnector < SourceConnector
3030
source_defined_primary_key: [["element_id"]]
3131
}.freeze
3232

33-
# Commands for unstructured data operations
33+
# Data types
3434
UNSTRUCTURED = "unstructured"
35+
SEMISTRUCTURED = "semistructured"
36+
37+
# Commands for unstructured & semi-structured data operations
3538
LIST_FILES_CMD = "list_files"
3639
DOWNLOAD_FILE_CMD = "download_file"
3740

3841
def unstructured_data?(connection_config)
3942
connection_config["data_type"] == UNSTRUCTURED
4043
end
4144

45+
def semistructured_data?(connection_config)
46+
connection_config["data_type"] == SEMISTRUCTURED
47+
end
48+
4249
def create_unstructured_stream
4350
Multiwoven::Integrations::Protocol::Stream.new(
4451
name: UNSTRUCTURED,
@@ -47,6 +54,15 @@ def create_unstructured_stream
4754
**UNSTRUCTURED_STREAM_CONFIG
4855
)
4956
end
57+
58+
def create_semistructured_stream
59+
Multiwoven::Integrations::Protocol::Stream.new(
60+
name: SEMISTRUCTURED,
61+
action: StreamAction["fetch"],
62+
json_schema: UNSTRUCTURED_SCHEMA,
63+
**UNSTRUCTURED_STREAM_CONFIG
64+
)
65+
end
5066
end
5167
end
5268
end

integrations/lib/multiwoven/integrations/rollout.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
module Multiwoven
44
module Integrations
5-
VERSION = "0.34.8"
5+
VERSION = "0.34.9"
66

77
ENABLED_SOURCES = %w[
88
Snowflake

0 commit comments

Comments
 (0)