This repository was archived by the owner on Sep 23, 2024. It is now read-only.
This repository was archived by the owner on Sep 23, 2024. It is now read-only.
KeyError: 'type' when loading data from tap-salesforce #357
Description
Describe the bug
target-snowflake fails with KeyError: 'type' if loading data coming from tap-salesforce.
It does not happen always - the data structures from Salesforce must contain some pattern, not sure which one.
To Reproduce
Not sure, what data from Salesforce causes the issue. It does not fail with my development sandbox env, but it happens with my production Salesforce environment.
Expected behavior
target does not fail.
Screenshots
Example Meltano log:
2023-03-03T23:05:50.416866Z [info ] INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.12344241142272949, "tags": {"endpoint": "batch_result_list", "sobject": "Contact", "status": "succeeded"}} cmd_type=elb consumer=False name=tap-salesforce producer=True stdio=stderr string_id=tap-salesforce
2023-03-03T23:05:50.418542Z [info ] INFO Making GET request to https://gooddata.my.salesforce.com/services/async/53.0/job/7506R00000dQHFBQA4/batch/7516R00000panqmQAA/result/7526R00000S2vi7 with params: None cmd_type=elb consumer=False name=tap-salesforce producer=True stdio=stderr string_id=tap-salesforce
2023-03-03T23:05:52.104308Z [info ] INFO METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"endpoint": "LeadHistory"}} cmd_type=elb consumer=False name=tap-salesforce producer=True stdio=stderr string_id=tap-salesforce
2023-03-03T23:05:52.109644Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.109913Z [info ] File "/home/jacek/work/src/panther-internal-analytics/data_pipeline/.meltano/loaders/target-snowflake/venv/bin/target-snowflake", line 8, in <module> cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.110175Z [info ] sys.exit(main()) cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.110322Z [info ] File "/home/jacek/work/src/panther-internal-analytics/data_pipeline/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/target_snowflake/__init__.py", line 526, in main cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.110446Z [info ] persist_lines(config, singer_messages, table_cache, file_format_type) cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.110604Z [info ] File "/home/jacek/work/src/panther-internal-analytics/data_pipeline/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/target_snowflake/__init__.py", line 143, in persist_lines cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.110725Z [info ] stream_utils.adjust_timestamps_in_record(o['record'], schemas[stream]) cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.110852Z [info ] File "/home/jacek/work/src/panther-internal-analytics/data_pipeline/.meltano/loaders/target-snowflake/venv/lib/python3.10/site-packages/target_snowflake/stream_utils.py", line 69, in adjust_timestamps_in_record cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.110973Z [info ] if 'string' in schema['properties'][key]['type'] and \ cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.111089Z [info ] KeyError: 'type' cmd_type=elb consumer=True name=target-snowflake-sfdc producer=False stdio=stderr string_id=target-snowflake-sfdc
2023-03-03T23:05:52.401696Z [error ]
Your environment
- Version of target: latest
- Version of python 3.10
Additional context
N/A