Skip to content

Commit 911771f

Browse files
atttiwariprijendev
atttiwari
and
prijendev
authored
Added unit and integration test cases (#8)
* SAC-29908: Added unit and integration test cases * SAC-29908: fix integration test case * Reshuffle aws env downloads * SAC-29908: Updated config.yaml to fix circle ci issue * SAC-29908: Updated config.yaml to fix circle ci issue * SAC-29908: Updated config.yaml to fix circle ci issue * SAC-29908: Updated versions for some streams * SAC-29908: Updated codebase for updated api endpoints and schema * SAC-29908: Fixed integration test issues * SAC-29908: Fixed unit test issues * SAC-29908: Fixed integration test issues --------- Co-authored-by: Prijen Khokhani <88327452+prijendev@users.noreply.github.com>
1 parent 062968f commit 911771f

27 files changed

+1427
-122
lines changed

‎.circleci/config.yml‎

Lines changed: 22 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -24,31 +24,28 @@ jobs:
2424
source /usr/local/share/virtualenvs/tap-ms-teams/bin/activate
2525
uv pip install pylint
2626
pylint tap_ms_teams -d C,R,W
27-
# - run:
28-
# name: "Unit Tests"
29-
# command: |
30-
# source /usr/local/share/virtualenvs/tap-ms-teams/bin/activate
31-
# uv pip install pytest coverage parameterized
32-
# coverage run -m pytest tests/unittests
33-
# coverage html
34-
# when: always
35-
# - store_test_results:
36-
# path: test_output/report.xml
37-
# - store_artifacts:
38-
# path: htmlcov
39-
# - run:
40-
# name: 'Integration Tests'
41-
# command: |
42-
# aws s3 cp s3://com-stitchdata-dev-deployment-assets/environments/tap-tester/sandbox dev_env.sh
43-
# source dev_env.sh
44-
# source /usr/local/share/virtualenvs/tap-tester/bin/activate
45-
# run-test --tap=tap-ms-teams \
46-
# --target=target-stitch \
47-
# --orchestrator=stitch-orchestrator \
48-
# --email=harrison+sandboxtest@stitchdata.com \
49-
# --password=$SANDBOX_PASSWORD \
50-
# --client-id=50 \
51-
# tests
27+
- run:
28+
name: "Unit Tests"
29+
command: |
30+
source /usr/local/share/virtualenvs/tap-ms-teams/bin/activate
31+
uv pip install pytest coverage parameterized
32+
coverage run -m pytest tests/unittests
33+
coverage html
34+
when: always
35+
- store_test_results:
36+
path: test_output/report.xml
37+
- store_artifacts:
38+
path: htmlcov
39+
- run:
40+
name: 'Integration Tests'
41+
command: |
42+
source /usr/local/share/virtualenvs/tap-tester/bin/activate
43+
uv pip install --upgrade awscli
44+
aws s3 cp s3://com-stitchdata-dev-deployment-assets/environments/tap-tester/tap_tester_sandbox dev_env.sh
45+
source dev_env.sh
46+
unset USE_STITCH_BACKEND
47+
run-test --tap=tap-ms-teams tests
48+
5249
workflows:
5350
version: 2
5451
commit:

‎README.md‎

Lines changed: 22 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -11,31 +11,31 @@ This tap:
1111

1212
- Pulls raw data from the [Microsoft Graph API](https://docs.microsoft.com/en-us/graph/)
1313
- Extracts the following resources:
14-
- [users](https://docs.microsoft.com/en-us/graph/api/user-list?view=graph-rest-beta&tabs=http)
15-
- [groups](https://docs.microsoft.com/en-us/graph/teams-list-all-teams?context=graph%2Fapi%2Fbeta&view=graph-rest-beta)
14+
- [users](https://learn.microsoft.com/en-us/graph/api/user-list?view=graph-rest-1.0&tabs=http)
15+
- [groups](https://learn.microsoft.com/en-us/graph/teams-list-all-teams?context=graph%2Fapi&view=graph-rest-1.0)
1616
- [group_members](https://docs.microsoft.com/en-us/graph/api/group-list-members?view=graph-rest-1.0&tabs=http)
1717
- [group_owners](https://docs.microsoft.com/en-us/graph/api/group-list-owners?view=graph-rest-1.0&tabs=http)
1818
- [channels](https://docs.microsoft.com/en-us/graph/api/channel-list?view=graph-rest-1.0&tabs=http)
19-
- [channel_members](https://docs.microsoft.com/en-us/graph/api/conversationmember-list?view=graph-rest-beta&tabs=http)
20-
- [channel_tabs](https://docs.microsoft.com/en-us/graph/api/teamstab-list?view=graph-rest-beta)
21-
- [channel_messages](https://docs.microsoft.com/en-us/graph/api/chatmessage-delta?view=graph-rest-beta&tabs=http)
22-
- [channel_message_replies](https://docs.microsoft.com/en-us/graph/api/channel-list-messagereplies?view=graph-rest-beta&tabs=http)
23-
- [conversations](https://docs.microsoft.com/en-us/graph/api/group-list-conversations?view=graph-rest-beta&tabs=http)
24-
- [conversation_threads](https://docs.microsoft.com/en-us/graph/api/conversation-list-threads?view=graph-rest-beta&tabs=http)
25-
- [conversation_posts](https://docs.microsoft.com/en-us/graph/api/conversationthread-list-posts?view=graph-rest-beta&tabs=http)
26-
- [team_drives](https://docs.microsoft.com/en-us/graph/api/drive-get?view=graph-rest-beta&tabs=http#get-the-document-library-associated-with-a-group)
27-
- [team_device_usage_report](https://docs.microsoft.com/en-us/graph/api/reportroot-getteamsdeviceusageuserdetail?view=graph-rest-beta)
19+
- [channel_members](https://learn.microsoft.com/en-us/graph/api/channel-list-members?view=graph-rest-1.0&tabs=http)
20+
- [channel_tabs](https://learn.microsoft.com/en-us/graph/api/channel-list-tabs?view=graph-rest-1.0&tabs=http)
21+
- [channel_messages](https://learn.microsoft.com/en-us/graph/api/channel-list-messages?view=graph-rest-1.0&tabs=http)
22+
- [channel_message_replies](https://learn.microsoft.com/en-us/graph/api/chatmessage-list-replies?view=graph-rest-1.0&tabs=http)
23+
- [conversations](https://learn.microsoft.com/en-us/graph/api/group-list-conversations?view=graph-rest-1.0&tabs=http)
24+
- [conversation_threads](https://learn.microsoft.com/en-us/graph/api/conversation-list-threads?view=graph-rest-1.0&tabs=http)
25+
- [conversation_posts](https://learn.microsoft.com/en-us/graph/api/conversationthread-list-posts?view=graph-rest-1.0&tabs=http)
26+
- [team_drives](https://learn.microsoft.com/en-us/graph/api/drive-get?view=graph-rest-1.0&tabs=http#get-the-document-library-associated-with-a-group)
27+
- [team_device_usage_report](https://learn.microsoft.com/en-us/graph/api/reportroot-getteamsdeviceusageuserdetail?view=graph-rest-1.0&tabs=http)
2828
- Outputs the schema for each resource
2929
- Incrementally pulls data based on the input state
3030

3131
## Streams
3232

33-
- [users](https://docs.microsoft.com/en-us/graph/api/user-list?view=graph-rest-beta&tabs=http)
33+
- [users](https://learn.microsoft.com/en-us/graph/api/user-list?view=graph-rest-1.0&tabs=http)
3434
- Data key: value
3535
- Primary keys: id
3636
- Replication strategy: Full Table
3737
- Transformations: camelCase to snake_case
38-
- [groups](https://docs.microsoft.com/en-us/graph/teams-list-all-teams?context=graph%2Fapi%2Fbeta&view=graph-rest-beta)
38+
- [groups](https://learn.microsoft.com/en-us/graph/teams-list-all-teams?context=graph%2Fapi&view=graph-rest-1.0)
3939
- Data key: value
4040
- Primary keys: id
4141
- Replication strategy: Full Table
@@ -55,53 +55,53 @@ This tap:
5555
- Primary keys: id
5656
- Replication strategy: Full Table
5757
- Transformations: camelCase to snake_case
58-
- [channel_members](https://docs.microsoft.com/en-us/graph/api/conversationmember-list?view=graph-rest-beta&tabs=http)
58+
- [channel_members](https://learn.microsoft.com/en-us/graph/api/channel-list-members?view=graph-rest-1.0&tabs=http)
5959
- Data key: value
6060
- Primary keys: id
6161
- Replication strategy: Full Table
6262
- Transformations: camelCase to snake_case
63-
- [channel_tabs](https://docs.microsoft.com/en-us/graph/api/teamstab-list?view=graph-rest-beta)
63+
- [channel_tabs](https://learn.microsoft.com/en-us/graph/api/channel-list-tabs?view=graph-rest-1.0&tabs=http)
6464
- Data key: value
6565
- Primary keys: id
6666
- Replication strategy: Full Table
6767
- Transformations: camelCase to snake_case
68-
- [channel_messages](https://docs.microsoft.com/en-us/graph/api/chatmessage-delta?view=graph-rest-beta&tabs=http)
68+
- [channel_messages](https://learn.microsoft.com/en-us/graph/api/channel-list-messages?view=graph-rest-1.0&tabs=http)
6969
- Data key: value
7070
- Primary keys: id
7171
- Replication strategy: Incremental (query all, filter results)
7272
- Bookmark: ucreatedDateTime OR lastModifiedDateTime OR deletedDateTime
7373
- Transformations: camelCase to snake_case
74-
- [channel_message_replies](https://docs.microsoft.com/en-us/graph/api/channel-list-messagereplies?view=graph-rest-beta&tabs=http)
74+
- [channel_message_replies](https://learn.microsoft.com/en-us/graph/api/chatmessage-list-replies?view=graph-rest-1.0&tabs=http)
7575
- Data key: value
7676
- Primary keys: id
7777
- Replication strategy: Incremental (query all, filter results)
7878
- Bookmark: ucreatedDateTime OR lastModifiedDateTime OR deletedDateTime
7979
- Transformations: camelCase to snake_case
80-
- [conversations](https://docs.microsoft.com/en-us/graph/api/group-list-conversations?view=graph-rest-beta&tabs=http)
80+
- [conversations](https://learn.microsoft.com/en-us/graph/api/group-list-conversations?view=graph-rest-1.0&tabs=http)
8181
- Data key: value
8282
- Primary keys: id
8383
- Replication strategy: Incremental (query all, filter results)
8484
- Bookmark: lastDeliveredDateTime
8585
- Transformations: camelCase to snake_case
86-
- [conversation_threads](https://docs.microsoft.com/en-us/graph/api/conversation-list-threads?view=graph-rest-beta&tabs=http)
86+
- [conversation_threads](https://learn.microsoft.com/en-us/graph/api/conversation-list-threads?view=graph-rest-1.0&tabs=http)
8787
- Data key: value
8888
- Primary keys: id
8989
- Replication strategy: Incremental (query all, filter results)
9090
- Bookmark: lastDeliveredDateTime
9191
- Transformations: camelCase to snake_case
92-
- [conversation_posts](https://docs.microsoft.com/en-us/graph/api/conversationthread-list-posts?view=graph-rest-beta&tabs=http)
92+
- [conversation_posts](https://learn.microsoft.com/en-us/graph/api/conversationthread-list-posts?view=graph-rest-1.0&tabs=http)
9393
- Data key: value
9494
- Primary keys: id
9595
- Replication strategy: Incremental (query all, filter results)
9696
- Bookmark: lastDeliveredDateTime
9797
- Transformations: camelCase to snake_case
98-
- [team_drives](https://docs.microsoft.com/en-us/graph/api/drive-get?view=graph-rest-beta&tabs=http#get-the-document-library-associated-with-a-group)
98+
- [team_drives](https://learn.microsoft.com/en-us/graph/api/drive-get?view=graph-rest-1.0&tabs=http#get-the-document-library-associated-with-a-group)
9999
- Data key: value
100100
- Primary keys: id
101101
- Replication strategy: Incremental (query all, filter results)
102102
- Bookmark: lastModifiedDateTime
103103
- Transformations: camelCase to snake_case
104-
- [team_device_usage_report](https://docs.microsoft.com/en-us/graph/api/reportroot-getteamsdeviceusageuserdetail?view=graph-rest-beta)
104+
- [team_device_usage_report](https://learn.microsoft.com/en-us/graph/api/reportroot-getteamsdeviceusageuserdetail?view=graph-rest-1.0&tabs=http)
105105
- Primary keys: id
106106
- Replication strategy: Incremental (query all, filter results)
107107
- Bookmark: report_refresh_date
@@ -161,7 +161,6 @@ The Microsoft Graph API uses OAuth for authentication. Follow these instructions
161161
10. Team > Team.ReadBasic.All
162162
11. User > User.Read.All
163163
16. Click Add permissions
164-
165164

166165
## Quick Start
167166

‎tap_ms_teams/__init__.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ def sync(client, config, catalog, state):
4646
stream.write_schema()
4747
stream_metadata = metadata.to_map(catalog_entry.metadata)
4848

49-
bookmark_date = stream.get_bookmark(stream.name,
49+
bookmark_date = stream.get_bookmark(stream.name, stream.replication_key,
5050
config['start_date'])
5151
bookmark_dttm = strptime_to_utc(bookmark_date)
5252
max_bookmark_value = None
@@ -73,6 +73,11 @@ def sync(client, config, catalog, state):
7373

7474
record_timestamp = stream.max_from_replication_dates(
7575
record)
76+
if record_timestamp is None:
77+
LOGGER.info(
78+
'Stream: %s - record missing replication timestamps, skipping',
79+
stream.name)
80+
continue
7681
if record_timestamp > max_bookmark_dttm:
7782
max_bookmark_value = strftime(record_timestamp)
7883

@@ -85,7 +90,7 @@ def sync(client, config, catalog, state):
8590
stream_metadata,
8691
))
8792
counter.increment()
88-
stream.update_bookmark(stream.name, max_bookmark_value)
93+
stream.update_bookmark(stream.name, stream.replication_key, max_bookmark_value)
8994
stream.write_state()
9095
stream.update_currently_syncing(None)
9196
stream.write_state()

‎tap_ms_teams/catalog.py‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,35 @@
1-
import singer
1+
from singer import metadata
22

33
def generate_catalog(streams):
44

55
catalog = {}
66
catalog['streams'] = []
77
for stream in streams:
88
schema = stream.load_schema()
9+
10+
mdata = metadata.new()
11+
mdata = metadata.get_standard_metadata(
12+
schema=schema,
13+
key_properties=getattr(stream, "key_properties"),
14+
valid_replication_keys=(getattr(stream, "valid_replication_keys") or []),
15+
replication_method=getattr(stream, "replication_method")
16+
)
17+
mdata = metadata.to_map(mdata)
18+
19+
automatic_keys = getattr(stream, "valid_replication_keys") or []
20+
for field_name in schema.get("properties", {}).keys():
21+
if field_name in automatic_keys:
22+
mdata = metadata.write(
23+
mdata, ("properties", field_name), "inclusion", "automatic"
24+
)
25+
26+
mdata = metadata.to_list(mdata)
27+
928
catalog_entry = {
1029
'stream': stream.name,
1130
'tap_stream_id': stream.name,
1231
'schema': schema,
13-
'metadata': singer.metadata.get_standard_metadata(
14-
schema=schema,
15-
key_properties=stream.key_properties,
16-
valid_replication_keys=stream.valid_replication_keys,
17-
replication_method=stream.replication_method)
32+
'metadata': mdata
1833
}
1934
catalog['streams'].append(catalog_entry)
2035

‎tap_ms_teams/client.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -179,13 +179,11 @@ def make_request(self, method, url=None, params=None, data=None):
179179
LOGGER.info("Making %s request to %s with params: %s", method, url, params)
180180
response = self.session.get(url, headers=headers, allow_redirects=True)
181181
elif method == "POST":
182-
LOGGER.info("Making %s request to %s with body %s", method, url, data)
182+
LOGGER.info("Making %s request to %s", method, url)
183183
response = self.session.post(url, data=data)
184184
else:
185185
raise Exception("Unsupported HTTP method")
186186

187-
LOGGER.info("Received code: %s", response.status_code)
188-
189187
if response.status_code == 401:
190188
LOGGER.info(
191189
"Received unauthorized error code, retrying: %s", response.text)

‎tap_ms_teams/schemas/channel_members.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@
2828
},
2929
"channel_id": {
3030
"type": ["null", "string"]
31+
},
32+
"group_id": {
33+
"type": ["null", "string"]
3134
}
3235
}
3336
}

‎tap_ms_teams/schemas/channel_message_replies.json‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,15 @@
55
"id": {
66
"type": ["null", "string"]
77
},
8+
"group_id": {
9+
"type": ["null", "string"]
10+
},
11+
"channel_id": {
12+
"type": ["null", "string"]
13+
},
14+
"message_id": {
15+
"type": ["null", "string"]
16+
},
817
"reply_to_id": {
918
"type": ["null", "string"]
1019
},
@@ -93,7 +102,7 @@
93102
"type": ["null", "object"],
94103
"additionalProperties": true,
95104
"properties": {
96-
"teamId": {
105+
"team_id": {
97106
"type": ["null", "string"]
98107
},
99108
"channel_id": {

‎tap_ms_teams/schemas/channel_messages.json‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,12 @@
55
"id": {
66
"type": ["null", "string"]
77
},
8+
"group_id": {
9+
"type": ["null", "string"]
10+
},
11+
"channel_id": {
12+
"type": ["null", "string"]
13+
},
814
"reply_to_id": {
915
"type": ["null", "string"]
1016
},
@@ -18,7 +24,7 @@
1824
"type": ["null", "string"],
1925
"format": "date-time"
2026
},
21-
"lastModified_date_time": {
27+
"last_modified_date_time": {
2228
"type": ["null", "string"],
2329
"format": "date-time"
2430
},
@@ -93,7 +99,7 @@
9399
"type": ["null", "object"],
94100
"additionalProperties": true,
95101
"properties": {
96-
"teamId": {
102+
"team_id": {
97103
"type": ["null", "string"]
98104
},
99105
"channel_id": {
@@ -170,7 +176,7 @@
170176
"display_name": {
171177
"type": ["null", "string"]
172178
},
173-
"userIdentity_type": {
179+
"user_identity_type": {
174180
"type": ["null", "string"]
175181
}
176182
}

‎tap_ms_teams/schemas/channels.json‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,29 @@
55
"id": {
66
"type": ["null", "string"]
77
},
8+
"group_id": {
9+
"type": ["null", "string"]
10+
},
811
"display_name": {
912
"type": ["null", "string"]
1013
},
1114
"description": {
1215
"type": ["null", "string"]
1316
},
17+
"is_favorite_by_default": {
18+
"type": ["null", "boolean"]
19+
},
1420
"email":{
1521
"type": ["null", "string"]
1622
},
1723
"web_url":{
1824
"type": ["null", "string"]
25+
},
26+
"membership_type": {
27+
"type": ["null", "string"]
28+
},
29+
"is_archived": {
30+
"type": ["null", "boolean"]
1931
}
2032
}
2133
}

‎tap_ms_teams/schemas/conversation_posts.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@
55
"id": {
66
"type": ["null", "string"]
77
},
8+
"group_id": {
9+
"type": ["null", "string"]
10+
},
811
"thread_id": {
912
"type": ["null", "string"]
1013
},

0 commit comments

Comments
 (0)