Skip to content

Commit d3d94df

Browse files
authored
chore: update descriptions and examples of complex fields (#661)
The titles, descriptions, and examples of some of the more complex fields in incremental sync and authenticator components were lacking a bit. This PR updates them to add more detail and examples, and have more professional-looking titles, as these are now all used to power the labels and descriptions we show in the Builder UI. This will help users when configuring these fields in the UI. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Improved schema and model documentation for datetime-related fields, including detailed descriptions and examples for datetime formats and ISO8601 durations. * Clarified default values and added illustrative examples for datetime properties, including current UTC time formatting. * Enhanced descriptions and titles for several boolean options and pagination properties to improve clarity and consistency. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent 019c027 commit d3d94df

File tree

2 files changed

+114
-33
lines changed

2 files changed

+114
-33
lines changed

airbyte_cdk/sources/declarative/declarative_component_schema.yaml

Lines changed: 82 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -940,14 +940,53 @@ definitions:
940940
- "{{ config['record_cursor'] }}"
941941
cursor_datetime_formats:
942942
title: Cursor Datetime Formats
943-
description: The possible formats for the cursor field, in order of preference. The first format that matches the cursor field value will be used to parse it. If not provided, the `datetime_format` will be used.
944943
type: array
945944
items:
946945
type: string
947-
examples:
948-
- "%Y-%m-%dT%H:%M:%S.%f%z"
949-
- "%Y-%m-%d"
950-
- "%s"
946+
description: |
947+
The possible formats for the cursor field, in order of preference. The first format that matches the cursor field value will be used to parse it. If not provided, the Outgoing Datetime Format will be used.
948+
Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available:
949+
* **%s**: Epoch unix timestamp - `1686218963`
950+
* **%s_as_float**: Epoch unix timestamp in seconds as float with microsecond precision - `1686218963.123456`
951+
* **%ms**: Epoch unix timestamp - `1686218963123`
952+
* **%a**: Weekday (abbreviated) - `Sun`
953+
* **%A**: Weekday (full) - `Sunday`
954+
* **%w**: Weekday (decimal) - `0` (Sunday), `6` (Saturday)
955+
* **%d**: Day of the month (zero-padded) - `01`, `02`, ..., `31`
956+
* **%b**: Month (abbreviated) - `Jan`
957+
* **%B**: Month (full) - `January`
958+
* **%m**: Month (zero-padded) - `01`, `02`, ..., `12`
959+
* **%y**: Year (without century, zero-padded) - `00`, `01`, ..., `99`
960+
* **%Y**: Year (with century) - `0001`, `0002`, ..., `9999`
961+
* **%H**: Hour (24-hour, zero-padded) - `00`, `01`, ..., `23`
962+
* **%I**: Hour (12-hour, zero-padded) - `01`, `02`, ..., `12`
963+
* **%p**: AM/PM indicator
964+
* **%M**: Minute (zero-padded) - `00`, `01`, ..., `59`
965+
* **%S**: Second (zero-padded) - `00`, `01`, ..., `59`
966+
* **%f**: Microsecond (zero-padded to 6 digits) - `000000`, `000001`, ..., `999999`
967+
* **%_ms**: Millisecond (zero-padded to 3 digits) - `000`, `001`, ..., `999`
968+
* **%z**: UTC offset - `(empty)`, `+0000`, `-04:00`
969+
* **%Z**: Time zone name - `(empty)`, `UTC`, `GMT`
970+
* **%j**: Day of the year (zero-padded) - `001`, `002`, ..., `366`
971+
* **%U**: Week number of the year (Sunday as first day) - `00`, `01`, ..., `53`
972+
* **%W**: Week number of the year (Monday as first day) - `00`, `01`, ..., `53`
973+
* **%c**: Date and time representation - `Tue Aug 16 21:30:00 1988`
974+
* **%x**: Date representation - `08/16/1988`
975+
* **%X**: Time representation - `21:30:00`
976+
* **%%**: Literal '%' character
977+
978+
Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the [Python documentation](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
979+
examples:
980+
- "%Y-%m-%d"
981+
- "%Y-%m-%d %H:%M:%S"
982+
- "%Y-%m-%dT%H:%M:%S"
983+
- "%Y-%m-%dT%H:%M:%SZ"
984+
- "%Y-%m-%dT%H:%M:%S%z"
985+
- "%Y-%m-%dT%H:%M:%S.%fZ"
986+
- "%Y-%m-%dT%H:%M:%S.%f%z"
987+
- "%Y-%m-%d %H:%M:%S.%f+00:00"
988+
- "%s"
989+
- "%ms"
951990
start_datetime:
952991
title: Start Datetime
953992
description: The datetime that determines the earliest record that should be synced.
@@ -1024,33 +1063,45 @@ definitions:
10241063
- "%s_as_float"
10251064
cursor_granularity:
10261065
title: Cursor Granularity
1027-
description:
1066+
description: |
10281067
Smallest increment the datetime_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one, e.g. for %Y-%m-%d the granularity should
10291068
be P1D, for %Y-%m-%dT%H:%M:%SZ the granularity should be PT1S. Given this field is provided, `step` needs to be provided as well.
1069+
* **PT0.000001S**: 1 microsecond
1070+
* **PT0.001S**: 1 millisecond
1071+
* **PT1S**: 1 second
1072+
* **PT1M**: 1 minute
1073+
* **PT1H**: 1 hour
1074+
* **P1D**: 1 day
10301075
type: string
10311076
examples:
10321077
- "PT1S"
10331078
is_data_feed:
1034-
title: Whether the target API is formatted as a data feed
1079+
title: Data Feed API
10351080
description: A data feed API is an API that does not allow filtering and paginates the content from the most recent to the least recent. Given this, the CDK needs to know when to stop paginating and this field will generate a stop condition for pagination.
10361081
type: boolean
10371082
is_client_side_incremental:
1038-
title: Whether the target API does not support filtering and returns all data (the cursor filters records in the client instead of the API side)
1039-
description: If the target API endpoint does not take cursor values to filter records and returns all records anyway, the connector with this cursor will filter out records locally, and only emit new records from the last sync, hence incremental. This means that all records would be read from the API, but only new records will be emitted to the destination.
1083+
title: Client-side Incremental Filtering
1084+
description: Set to True if the target API endpoint does not take cursor values to filter records and returns all records anyway. This will cause the connector to filter out records locally, and only emit new records from the last sync, hence incremental. This means that all records would be read from the API, but only new records will be emitted to the destination.
10401085
type: boolean
10411086
is_compare_strictly:
1042-
title: Whether to skip requests if the start time equals the end time
1043-
description: Set to True if the target API does not accept queries where the start time equal the end time.
1087+
title: Strict Start-End Time Comparison
1088+
description: Set to True if the target API does not accept queries where the start time equal the end time. This will cause those requests to be skipped.
10441089
type: boolean
10451090
default: False
10461091
global_substream_cursor:
1047-
title: Whether to store cursor as one value instead of per partition
1048-
description: This setting optimizes performance when the parent stream has thousands of partitions by storing the cursor as a single value rather than per partition. Notably, the substream state is updated only at the end of the sync, which helps prevent data loss in case of a sync failure. See more info in the [docs](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/incremental-syncs).
1092+
title: Global Substream Cursor
1093+
description: Setting to True causes the connector to store the cursor as one value, instead of per-partition. This setting optimizes performance when the parent stream has thousands of partitions. Notably, the substream state is updated only at the end of the sync, which helps prevent data loss in case of a sync failure. See more info in the [docs](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/incremental-syncs).
10491094
type: boolean
10501095
default: false
10511096
lookback_window:
10521097
title: Lookback Window
1053-
description: Time interval before the start_datetime to read data for, e.g. P1M for looking back one month.
1098+
description: |
1099+
Time interval (ISO8601 duration) before the start_datetime to read data for, e.g. P1M for looking back one month.
1100+
* **PT1H**: 1 hour
1101+
* **P1D**: 1 day
1102+
* **P1W**: 1 week
1103+
* **P1M**: 1 month
1104+
* **P1Y**: 1 year
10541105
type: string
10551106
interpolation_context:
10561107
- config
@@ -1071,7 +1122,13 @@ definitions:
10711122
- "starting_time"
10721123
step:
10731124
title: Step
1074-
description: The size of the time window (ISO8601 duration). Given this field is provided, `cursor_granularity` needs to be provided as well.
1125+
description: |
1126+
The size of the time window (ISO8601 duration). Given this field is provided, `cursor_granularity` needs to be provided as well.
1127+
* **PT1H**: 1 hour
1128+
* **P1D**: 1 day
1129+
* **P1W**: 1 week
1130+
* **P1M**: 1 month
1131+
* **P1Y**: 1 year
10751132
type: string
10761133
examples:
10771134
- "P1W"
@@ -1911,7 +1968,13 @@ definitions:
19111968
type: string
19121969
expiration_duration:
19131970
title: Expiration Duration
1914-
description: The duration in ISO 8601 duration notation after which the session token expires, starting from the time it was obtained. Omitting it will result in the session token being refreshed for every request.
1971+
description: |
1972+
The duration in ISO 8601 duration notation after which the session token expires, starting from the time it was obtained. Omitting it will result in the session token being refreshed for every request.
1973+
* **PT1H**: 1 hour
1974+
* **P1D**: 1 day
1975+
* **P1W**: 1 week
1976+
* **P1M**: 1 month
1977+
* **P1Y**: 1 year
19151978
type: string
19161979
examples:
19171980
- "PT1H"
@@ -2693,6 +2756,7 @@ definitions:
26932756
- 2021-01-01
26942757
- 2021-01-01T00:00:00Z
26952758
- "{{ config['start_time'] }}"
2759+
- "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}"
26962760
datetime_format:
26972761
title: Datetime Format
26982762
description: |
@@ -3077,7 +3141,7 @@ definitions:
30773141
- 100
30783142
- "{{ config['page_size'] }}"
30793143
inject_on_first_request:
3080-
title: Inject Offset
3144+
title: Inject Offset on First Request
30813145
description: Using the `offset` with value `0` during the first request
30823146
type: boolean
30833147
default: false
@@ -3117,7 +3181,7 @@ definitions:
31173181
- 0
31183182
- 1
31193183
inject_on_first_request:
3120-
title: Inject Page Number
3184+
title: Inject Page Number on First Request
31213185
description: Using the `page number` with value defined by `start_from_page` during the first request
31223186
type: boolean
31233187
default: false

0 commit comments

Comments
 (0)